CN113902066A - A sentence-level sign language recognition method, system, device and terminal - Google Patents

Abstract

The invention belongs to the technical field of action recognition and discloses a sentence-level sign language recognition method, a system, equipment and a terminal, wherein the sentence-level sign language recognition method comprises the following steps: collecting original sign language data signals; preprocessing sign language data; dividing effective sign language activity part data; extracting effective characteristics; and (4) carrying out data training by using a random forest classifier to realize sign language recognition. The sign language recognition method of the invention adopts an effective feature extraction method, namely, a new feature is added on the basis of the original ten features, and the moderate degree of data fluctuation can be reflected, so that the method can effectively extract the sentence-level sign language signal features. The invention adopts a random forest classifier, wherein 50 decision trees are used, and the classifier determined by multiple experiments also has good effect of improving the accuracy of the hand recognition. The sign language identification method has good robustness, so that the method can be well applied to application scenes and has certain universality.

Description

A sentence-level sign language recognition method, system, device and terminal

technical field

The invention belongs to the technical field of action recognition, and in particular relates to a method, system, device and terminal for sentence-level sign language recognition.

Background technique

At present, although spoken language is the mainstream in today's world, it is undeniable that sign language is still important to the present invention. Especially in the era of artificial intelligence, it has a wide range of applications, such as smart homes, showrooms, and more. Sign language not only facilitates deaf people to integrate into intelligent life more quickly, but also facilitates the present invention to communicate with deaf people. For example, in a history museum, when deaf people need help, staff cannot communicate with them because they don't understand sign language. If a museum has a robot (or other interactive device) that can recognize and respond to the sign language of deaf people, they can help and serve them.

In recent years, research on sign language recognition has been mainly based on three categories: based on computer vision, based on wearable sensors, and based on wireless signals. Most studies have only focused on isolated word recognition, however it is difficult to use in everyday communication, where few can translate sentence-level sign language.

Existing computer vision-based sign language recognition systems use infrared light as a perception mechanism to translate sign language at the sentence level. Despite their high accuracy, privacy is a big concern, especially in private domain settings. Wearable sensor-based systems are invasive and require users to wear expensive equipment, which limits the user's freedom to some extent. Although wireless networks based on wireless signals (such as WIFI) do not violate human privacy and do not require any equipment to be worn, wireless networks are inconvenient to deploy and prone to interference, so they are not a good choice. And the sign language recognition system based on RFID technology has low technical cost and its tags are easy to deploy. RFID (Radio Frequency Identification) technology refers to wireless radio frequency identification technology.

Existing techniques have difficulty dividing and translating consecutive sign languages into words. In addition, there are differences between words and word-level words separated from sentences, for example, the word "you" is completely different from the word "you" in two sentences (where "you" is the subject and object). It is also a difficult problem to realize automatic segmentation of sign language. Previous studies on sign language usually extract sign language manually. Obviously, this is a huge workload.

To sum up, the existing sign language recognition systems generally have the following shortcomings: 1) sentence-level sign language recognition cannot be achieved; 2) equipment is not cheap. Therefore, there is an urgent need for a new sentence-level sign language recognition method and system to make up for the shortcomings of the prior art.

Through the above analysis, the existing problems and defects in the prior art are:

(1) Existing research on sign language recognition Most of the studies only focus on isolated word recognition, however, it is difficult to use in everyday communication, and few of them can translate sentence-level sign language.

(2) The existing computer vision-based sign language recognition system has insufficient privacy protection in the private domain environment; at the same time, the wearable sensor-based system is intrusive and requires users to wear expensive equipment, which limits the user's freedom to a certain extent. .

(3) The existing wireless network based on wireless signals is inconvenient to deploy, easy to be interfered, and not a good choice; the existing technology is difficult to divide continuous sign language and translate it into words, and it is also a difficult problem to realize automatic segmentation of sign language. The volume is huge and the equipment is not cheap.

The difficulty of solving the above problems and defects is as follows:

(1) Due to the continuity of the signal and the interference of noise, the extraction of effective sign language signals has become a significant challenge, which requires the design of reasonable and effective denoising methods and signal processing methods to achieve effective sign language data signal segmentation.

(2) At the same time, the use of radio frequency identification technology can effectively protect privacy and free users from wearable devices, but it is a problem to determine an effective sign language signal reception range, which requires a lot of experiments based on different ranges and in different scenarios. The following experiments are used to determine the reasonable acceptance range of sign language signals;

(3) Finally, the key lies in how to make the system distinguish different sign language signals, which requires the extraction of effective signal features, and after a lot of experiments and analysis, the key and effective signal features are finally determined to achieve the purpose of identifying sign language signals.

The significance of solving the above problems and defects is:

(1) The main contribution is the realization of sentence-level sign language signal recognition, which is different from the previous word-level sign language signal recognition, which significantly improves the efficiency of sign language communication and makes sign language communication faster and more convenient.

(2) At the same time, sign language segmentation based on radio frequency identification technology can effectively protect user privacy without wearing equipment, which greatly facilitates users.

(3) A reasonable deployment range can increase the accuracy of identification. In addition, radio frequency identification systems and passive tags significantly reduce application costs and have high commercial availability.

SUMMARY OF THE INVENTION

In view of the problems existing in the prior art, the present invention provides a sentence-level sign language recognition method, system, device and terminal, and in particular relates to a sentence-level sign language recognition method, system, device and terminal based on radio frequency identification RFID.

The present invention is implemented in this way, a statement-level sign language recognition method, the statement-level sign language recognition method comprising the following steps:

Step 1, a commercial radio frequency identification device based on the COST radio frequency identification technology acquires the phase sequence of a specific target between the identification system and the passive tag through radio, and obtains the original sign language data signal. In this step, the original sign language data signal is obtained, paving the way for further processing;

Step 2, perform phase offset correction and use threshold-based wavelet denoising method to remove phase offset caused by environmental hardware and Gaussian noise, and obtain a pure phase sequence. This step removes the noise in the original sign language data signal and improves the accuracy of sign language recognition;

Step 3, using the signal processing method based on the standard deviation to realize the segmentation of the effective sign language data signal. This step divides the effective sign language data signal and reduces the interference caused by the useless sign language data signal;

Step 4: By analyzing each feature between the same sign language signal and different sign language signals, select features that are stable in the same sign language signal but can be clearly distinguished between different sign language signals, and perform effective feature extraction. This step extracts the effective features for sign language recognition, which greatly improves the recognition accuracy;

The fifth step is to use the random forest classifier and use 50 decision trees to realize sign language recognition. This step finally achieves sentence-level sign language recognition.

Further, in step 2, the phase offset correction is performed and the threshold-based wavelet denoising method is used to remove the phase offset caused by the environmental hardware and Gaussian noise to obtain a pure phase sequence, including:

(1) Use the threshold-based wavelet denoising method to remove the offset caused by the environmental hardware and Gaussian noise, the formula is as follows:

θ _true = θ-θ _n (θ _n = πor2π);

Among them, n is the sequence number, θ _true is the real phase value after removing the phase offset, θ is the original phase sequence value, and θ _n is the phase offset value.

(2) The phase sequence after removing the noise by obtaining the sign language data signal from the API of the RFID reader is as follows:

θ={θ ₁ , θ ₂ ... θ _i ... θ _n };

where n is the number of phase sequences, θ is the phase sequence after noise removal, and θ _i is the phase value in the phase sequence.

(3) Using the same processing data to reduce the slight difference of the same gesture at different times, the calculation formula is as follows:

where S(i) is the normalized intermediate result, θ(i) is the phase value in the phase sequence, min(θ) is the minimum value in the phase sequence, max(θ) is the maximum value in the phase sequence, a (i) is the ith phase sequence sample after normalization.

Further, in step 3, the use of the standard deviation-based signal processing method to realize the segmentation of effective sign language data signals includes:

(1) Subtract the median from each phase value in the sample and find the absolute value:

m(i)=|a(i)-median(a)|;

in,

is the median, and m(i) is the absolute value of each number in the phase sequence a(i) minus the median median(a).

(2) Calculate the standard deviation by grouping the data in turn:

Among them, d(i) is the standard deviation of the new series after calculating the standard, k is the length of the series, and μ is the mean value series k=50.

(3) Using the value of d(i), a data group whose single data is greater than the threshold r (r=0.1) is valid data.

Further, in step 4, by analyzing each feature between the same sign language signal and different sign language signals, select and select the features that are stable in the same sign language signal and can be clearly distinguished between different sign language signals, and carry out effective analysis. Feature extraction, including:

(1) Ten eigenvalues of the obtained effective data set, including skewness, expected value, third-order center distance, mean, variance, standard deviation, kurtosis, energy, and the main frequency ratio to the maximum frequency peak.

(2) The SOS value of the extracted valid data:

SOS: Assuming n is the length of the array, starting from the first set of arrays, each number and the following (k-1) numbers form an array of length k, get the standard deviation value of each array, and get (N-k+ 1) standard deviations, find the sum of the (N-k+1) standard deviations and get SOS: .

Compare the fluctuation of the softness of the data; when the data fluctuates, the value is large; when the data fluctuation is small, it will become smaller, the formula is as follows:

where N is the total number of data in the dataset, μ is the average of k data, and k is the number of data per standard deviation.

Further, in step 5, the random forest classifier is used and 50 decision trees are used to realize sign language recognition, including:

(1) Disorganize the data and extract 1/10 group of data as the data test set, and the rest of the data as the training set, and import the test set and training set into the RF classifier for identification.

(2) Extract another 1/10 group of data as a data test set, and cyclically import the RF classifier until each group of data has been tested, and finally get the test result.

Another object of the present invention is to provide a statement-level sign language recognition system applying the statement-level sign language recognition method, the statement-level sign language recognition system comprising:

The sign language data signal acquisition module is used for commercial radio frequency identification devices based on COST radio frequency identification technology. It acquires the phase sequence of a specific target between the identification system and the passive tag through radio, and obtains the original sign language data signal;

The signal denoising processing module is used to correct the phase offset and use the threshold-based wavelet denoising method to remove the phase offset caused by the environmental hardware and Gaussian noise, and obtain a pure phase sequence;

The sign language signal segmentation module is used to realize the segmentation of effective sign language data signals by using the signal processing method based on standard deviation;

The feature extraction module is used to select features that are stable in the same sign language signal and can be clearly distinguished between different sign language signals by analyzing each feature between the same sign language signal and different sign language signals, and perform effective feature extraction. ;

A sign language recognition module for sign language recognition using 50 decision trees using a random forest classifier.

Another object of the present invention is to provide a computer device, the computer device includes a memory and a processor, the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the following step:

The commercial RFID device based on COST RFID technology acquires the phase sequence of a specific target between the identification system and the passive tag by radio, and obtains the original sign language data signal; performs phase offset correction and uses threshold-based wavelet denoising method Remove the phase offset caused by environmental hardware and Gaussian noise to obtain a pure phase sequence; use the signal processing method based on standard deviation to achieve effective sign language data signal segmentation;

By analyzing each feature between the same sign language signal and different sign language signals, select and select the features that are stable in the same sign language signal and can be clearly distinguished between different sign language signals to perform effective feature extraction; use random forest classifiers , and use 50 decision trees for sign language recognition.

Another object of the present invention is to provide a computer-readable storage medium, which stores a computer program, and when the computer program is executed by a processor, causes the processor to perform the following steps:

The commercial RFID device based on COST RFID technology acquires the phase sequence of a specific target between the identification system and the passive tag by radio, and obtains the original sign language data signal; performs phase offset correction and uses threshold-based wavelet denoising method Remove the phase offset caused by environmental hardware and Gaussian noise to obtain a pure phase sequence; use the signal processing method based on standard deviation to achieve effective sign language data signal segmentation;

By analyzing each feature between the same sign language signal and different sign language signals, select and select the features that are stable in the same sign language signal and can be clearly distinguished between different sign language signals to perform effective feature extraction; use random forest classifiers , and use 50 decision trees for sign language recognition.

Another object of the present invention is to provide an information data processing terminal for implementing the sentence-level sign language recognition system.

Another object of the present invention is to provide an application of the sentence-level sign language recognition system in the field of artificial intelligence technology.

Combined with all the above technical solutions, the advantages and positive effects of the present invention are: the sentence-level sign language recognition method provided by the present invention obtains relatively pure phase characteristics by collecting the signal phase sequence received by commercial RFID, and gives A method for realizing sign language segmentation, which can be used for sign language recognition based on sign language action waveforms and sign language information exchange between humans and computers. At the same time, effective feature extraction and classifier selection are the keys to sign language recognition, so the present invention has finally determined 11 effective features after many comparisons. The present invention bridges the gap between corresponding low-cost sentence-level sign language recognition by evaluating the system in a real language environment.

The RFID-based sign language recognition method provided by the present invention can realize sentence-level sign language recognition under the condition of processing a large amount of data and complete-level sign language recognition tasks, which fills the defects of the sign language recognition system in RFID sentence-level recognition, and has a negative impact on the environment. The dynamic adaptability is good, and the economic cost is low. The sign language identification method of the present invention can realize sign language identification with only a small number of RFID passive tags and RFID readers, that is, the required economic cost is low, and the sign language identification can be realized without using expensive equipment.

The sign language recognition method of the present invention adopts an effective feature extraction method, that is, a new feature is added on the basis of the original ten features, namely SOS (Sum of Standard Deviations), which can reflect the mildness of data fluctuations, so The present invention can effectively extract sentence-level sign language signal features. The present invention adopts a random forest classifier in which 50 decision trees are used, and the classifier determined through many experiments also has a good effect of improving the accuracy of sign language recognition.

The present invention utilizes the signal phase sequence generated by the RFID system to perform sentence-level sign language recognition, breaks through the current RFID-based word-level sign language recognition, and has good practicability. The robustness of the sign language recognition method of the present invention is very good, so it can be well applied to application scenarios, and has certain universality.

Description of drawings

In order to illustrate the technical solutions of the embodiments of the present invention more clearly, the following will briefly introduce the accompanying drawings that need to be used in the embodiments of the present invention. Obviously, the drawings described below are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can also be obtained from these drawings without creative effort.

FIG. 1 is a flowchart of a sentence-level sign language recognition method provided by an embodiment of the present invention.

FIG. 2 is a schematic diagram of a sentence-level sign language recognition method provided by an embodiment of the present invention.

3 is a structural block diagram of a sentence-level sign language recognition system provided by an embodiment of the present invention;

In the figure: 1. Sign language data signal acquisition module; 2. Signal denoising processing module; 3. Sign language signal segmentation module; 4. Feature extraction module; 5. Sign language recognition module.

FIG. 4 is a schematic diagram of a sign language signal after phase calibration processing is performed on the sign language signal provided by an embodiment of the present invention.

FIG. 5 is a schematic diagram of a sign language signal provided by an embodiment of the present invention after the sign language signal is subjected to wavelet transform to remove noise caused by the environment.

FIG. 6 is a sample diagram of the phase sequence after normalizing the phase sequence and subtracting the median (a) of the phase sequence provided by an embodiment of the present invention.

FIG. 7 is a schematic diagram of calculating the standard deviation of a sample phase sequence and a sign language signal segmentation result after thresholding, according to an embodiment of the present invention.

FIG. 8 is an arrangement diagram of experimental equipment and a scene provided by an embodiment of the present invention.

FIG. 9 is a confusion matrix diagram of sign language recognition provided by an embodiment of the present invention.

FIG. 10 is a relationship diagram between the number of random forest decision trees and sign language recognition accuracy provided by an embodiment of the present invention.

FIG. 11 is an interference diagram of a sign language recognition system provided by dynamic multi-pathing according to an embodiment of the present invention.

FIG. 12 is a relationship diagram between the number of sign language recognitions and the recognition accuracy provided by an embodiment of the present invention.

Detailed ways

In order to make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention, but not to limit the present invention.

In view of the problems existing in the prior art, the present invention provides a sentence-level sign language recognition method, system, device and terminal. The present invention is described in detail below with reference to the accompanying drawings.

As shown in FIG. 1 , the sentence-level sign language recognition method provided by the embodiment of the present invention includes the following steps:

S101, a commercial radio frequency identification device based on COST radio frequency identification technology, obtains the phase sequence of a specific target between the identification system and the passive tag through radio, and obtains the original sign language data signal;

S102, performing phase offset correction and using a threshold-based wavelet denoising method to remove the phase offset caused by environmental hardware and Gaussian noise to obtain a pure phase sequence;

S103, using a signal processing method based on standard deviation to achieve effective sign language data signal segmentation;

S104, by analyzing each feature between the same sign language signal and different sign language signals, select and select features that are stable in the same sign language signal but can be clearly distinguished between different sign language signals, and perform effective feature extraction;

S105, using a random forest classifier and using 50 decision trees to realize sign language recognition.

The principle diagram of the sentence-level sign language recognition method provided by the embodiment of the present invention is shown in FIG. 2 .

As shown in FIG. 3 , the sentence-level sign language recognition system provided by the embodiment of the present invention includes:

Sign language data signal acquisition module 1, used for commercial radio frequency identification devices based on COST radio frequency identification technology, obtains the phase sequence of a specific target between the identification system and the passive tag through radio, and obtains the original sign language data signal;

The signal denoising processing module 2 is used to perform phase offset correction and use the threshold-based wavelet denoising method to remove the phase offset caused by environmental hardware and Gaussian noise, and obtain a pure phase sequence;

The sign language signal segmentation module 3 is used to realize the segmentation of effective sign language data signals by using the signal processing method based on standard deviation;

The feature extraction module 4 is used to select and select the features that are stable in the same sign language signal and can be clearly distinguished between different sign language signals by analyzing each feature between the same sign language signal and different sign language signals. extract;

The sign language recognition module 5 is used to realize sign language recognition using 50 decision trees by using the random forest classifier.

The technical solutions of the present invention will be further described below with reference to specific embodiments.

Example 1

Aiming at the above-mentioned defects and deficiencies in the prior art, the purpose of the present invention is to provide an RFID-based sign language recognition method, which can realize sentence-level sign language while processing a large amount of data and complete-level sign language recognition tasks. Recognition fills the shortcomings of sign language recognition system in RFID sentence-level recognition, has good dynamic adaptability to the environment, and has low economic cost.

To achieve the above object, the present invention adopts the following technical scheme:

The RFID-based sign language recognition method provided by the embodiment of the present invention specifically includes the following steps:

Step 1, the commercial radio frequency identification device based on the COST radio frequency identification technology obtains the phase sequence of the specific target between the identification system and the passive tag through radio, and obtains the original sign language data signal;

Step 2, perform phase offset correction and use threshold-based wavelet denoising method to remove phase offset caused by environmental hardware and Gaussian noise to obtain a pure phase sequence;

Step 3, using a signal processing method based on standard deviation to achieve effective sign language data signal segmentation;

Step 4, by analyzing each feature between the same sign language signal and different sign language signals, select and select the features that are stable in the same sign language signal but can be clearly distinguished between different sign language signals, and perform effective feature extraction;

The fifth step is to use the random forest classifier and use 50 decision trees to realize sign language recognition.

Specifically, the specific processing method of the step 2 includes:

Step 2.1: Use the threshold-based wavelet denoising method to remove the offset caused by the environmental hardware and Gaussian noise, the formula is as follows:

θ _true = θ-θ _i (θ _i =πor2π)

where i is the sequence number and θ is the phase.

Step 2.2: Use the same processing data to reduce the slight difference of the same gesture at different times. The calculation formula is as follows:

where S(i) is the normalized intermediate result, and a(i) is the ith phase sequence sample after normalization.

Specifically, the specific processing method of the step 3 includes:

Step 3.1: Subtract the median and absolute value from each phase value in the sample:

m(i)=|a(i)-median(a)|

where median(a) is the median;

Step 3.2: Calculate the standard deviation by grouping the data in sequence:

where d(i) is the standard deviation of the new series after calculating the standard, k is the length of the series, and μ is the mean series (k=50);

Step 3.3: Using the value of d(i), a data group whose single data is greater than the threshold r (r=0.1) is valid data.

Specifically, the specific processing method of the step 4 includes:

4.1: Ten eigenvalues of the extracted valid data set: skewness, expected value, third-order center distance, mean, variance, standard deviation, kurtosis, energy, main frequency ratio to the maximum frequency peak;

4.2: SOS value of the extracted valid data,

SOS: The present invention assumes that n is the length of the array. Starting with the first set of arrays, each number and the following (k-1) numbers form an array of length k. The present invention obtains the standard deviation value of each array, and then the present invention obtains (N-k+1) standard deviations, finds the sum, the sum of these (N-k+1) standard deviations, and obtains SOS;.

Function: You can compare the fluctuation of the softness of the data. When the data fluctuates, the value is large; when the data fluctuates less, it becomes small.

where N is the total number of data in the dataset, μ is the average of k data, k is the number of data per standard deviation, and k is taken as 5 in this experiment.

Specifically, the specific processing method of the step 5 includes:

Step 5.1: Disorganize the data and extract 1/10 of the data as the data test set, and the rest of the data as the training set, import the test set and the training set into the RF classifier for identification;

Step 5.2: Extract another 1/10 group of data as a data test set, and import the RF classifier cyclically until each group of data has been tested, and finally get the test result.

Example 2

An embodiment of the present invention provides a sign language recognition method, which can be applied to various system platforms, and the execution body of which can be a computer terminal or a processor of various mobile devices. The method flowchart of the method is shown in FIG. 1 . , including:

The RFID-based sign language recognition method of the present invention specifically includes the following steps:

Step 1: Collect the original sign language data signal

Commercial RFID devices based on COST RFID technology acquire the phase sequence of a specific target between the identification system and the passive tag by radio.

The RFID system consists of RFID tags, RFID readers and RFID directional antennas. Commercial RFID devices provide users with an application program interface. Through the API, users can obtain the received signal strength indication and passive tags to obtain the phase and other characteristics. API (ApplicationProgramInterface) is an application program interface, an application operation interface provided to the user.

Among these eigenvalues, the phase sequence of the signal has high stability and is rarely disturbed by the environment, so the present invention selects the phase sequence as the feature of the present invention for sign language recognition.

Step 2: De-noise the sign language signal

Perform phase offset correction and use threshold-based wavelet denoising method to remove the phase offset caused by environmental hardware and Gaussian noise, and obtain a pure phase sequence.

The phase sequence obtained from the RFID reader's API contains phase shifts due to hardware and Gaussian noise due to the environment.

Step 2.1: There are two types of phase shifts in the received signal, usually the phase sequence is shifted by π or 2π, as follows:

θ _true = θ-θ _i (θ _i =πor2π)

where i is the sequence number and θ is the phase. To calibrate the phase offset, the average of the resulting scent sequences was calculated and restored to the true phase sequence by adding or subtracting π or 2π, resulting in the phase sequence shown in Figure 4.

At the same time, in order to remove the Gaussian noise generated by the environment, the present invention continues to perform threshold-based discrete wavelet transform on the phase sequence obtained in the previous step to obtain the phase sequence shown in FIG. 5 .

Step 2.2: Obtain the sign language data signal from the API of the RFID reader through step 2.1. The phase sequence after noise removal is as follows:

θ={θ ₁ , θ ₂ ... θ _i ... θ _n } where n is the phase sequence number and θ(i) is the ith phase value.

At the same time, the same processing data is used to reduce the slight difference of the same gesture at different times. The calculation formula is as follows:

where S(i) is the normalized intermediate result, and a(i) is the ith phase sequence sample after normalization.

Step 3, using the signal processing method based on the standard deviation to realize the segmentation of the effective sign language data signal.

Step 3.1: Subtract the median and absolute value from each phase value in the sample:

m(i)=|a(i)-median(a)|

where median(a) is the median, and m(i) is the absolute value of the median minus the median from each number in the phase sequence a(i) to obtain the phase sequence diagram shown in Figure 6.

Step 3.2: Calculate the standard deviation by grouping the data in sequence:

where d(i) is the standard deviation of the new series after calculating the standard, k is the length of the series, and μ is the mean series (k=50);

Step 3.3: For the standard deviation d(i) after grouping, a data group with a single data greater than the threshold r (r=0.1) is regarded as valid data and extracted.

After the above steps, the present invention extracts an effective sign language data signal and realizes the segmentation of the sign language data, as shown in FIG. 7 .

Step 4: By analyzing each feature between the same sign language signal and different sign language signals, select features that are stable in the same sign language signal but can be clearly distinguished between different sign language signals, and perform effective feature extraction.

4.1: Ten eigenvalues of the extracted valid data set: skewness, expected value, third-order center distance, mean, variance, standard deviation, kurtosis, energy, main frequency ratio to the maximum frequency peak;

4.2: Introduce the extracted new feature SOS, namely Sum of Standard Deviations, which can measure the severity of data fluctuations.

SOS; the present invention assumes that n is the length of the array. Starting with the first set of arrays, each number and the following (k-1) numbers form an array of length k. The present invention obtains the standard deviation value of each array, and then the present invention obtains (N-k+1) standard deviations, finds the sum, the sum of these (N-k+1) standard deviations, and obtains SOS;.

Function: You can compare the fluctuation of the softness of the data. When the data fluctuates, the value is large; when the data fluctuates less, it becomes small.

where N is the total number of data in the dataset, μ is the average of k data, k is the number of data per standard deviation, and k is taken as 5 in this experiment.

The fifth step is to use the random forest classifier and use 50 decision trees to realize sign language recognition.

Step 5.1: Disorganize the data and extract 1/10 of the data as the data test set, and the rest of the data as the training set, import the test set and the training set into the RF classifier for identification;

Step 5.2: Extract another 1/10 group of data as a data test set, and import the RF classifier cyclically until each group of data has been tested, and finally get the test result.

Example 3

Experimental verification and evaluation are carried out in an indoor laboratory using a general-purpose RFID system. In this experiment, there are three different experiments in total to verify the recognition accuracy, robustness and advantages of the present invention compared with other methods. As shown in Figure 8, the main experimental equipment includes commercial RFID readers, the specific models used are Impinj R420 readers, common directional antennas and existing UHF passive tags, specifically AZ-9662RFID electronic tags. In addition, a network cable is used to connect the RFID reader to the computer, enabling it to send commands to send and receive data. This experiment recruited 8 volunteers, 5 males and 3 females, ranging in age from 19 to 29 years old, with a height distribution range of 1.56m-1.85m, and a weight distribution range of 46kg-70kg. The signal processing of sign language data during the experiment is based on MATLAB 2018b.

This example was carried out in a laboratory of approximately ^70m2 and including several tables and chairs. In order to determine the optimal distance between the antenna and the tag, the present invention tests the distance from 0.8m to 1.5m with a step size of 0.1m. When the distance between the two is more than 1.0m, the received sign language signal is chaotic, and it is difficult to extract the effective phase sequence features; when the distance between the two is less than 1.0m, the extracted phase sequence is difficult to reflect its characteristics. Therefore, the distance between the antenna and the tag is determined to be 1.0m. At the same time, the height of the device from the ground is set to 1.2m, which is suitable for most people. If the height is less than 1.2m, the interference of the ground to the signal will be greatly increased. As shown in Figure 8, the volunteer stood between the antenna and the tag.

The RFID-based sentence-level sign language recognition method provided by the embodiment of the present invention specifically includes the following steps:

Step 1, the commercial radio frequency identification device based on COST RFID technology obtains the original sign language data signal between the identification system and the passive tag through radio, extracts its phase sequence feature through the API of the RFID reader, and obtains the phase sequence of the original sign language data signal;

Step 2, use the phase sequence obtained in step 1 to perform phase offset correction and use threshold-based wavelet denoising method to remove Gaussian noise caused by the environment, and obtain a phase sequence after denoising;

Step 3, using the phase sequence obtained in step 2 after denoising, using a signal processing method based on standard deviation to segment the phase sequence of the valid sign language data signal, and obtain the phase sequence of the segmented valid sign language data;

Step 4: Using the phase sequence of the segmented valid sign language data obtained in step 3, by analyzing each feature between the phase sequence of the same sign language data and the phase sequences of different sign language data, select the phase sequence of the same sign language signal. The features that are stable and can be clearly distinguished between different sign language signal phase sequences can be effectively extracted to obtain eleven effective features;

Step 5: Use the eleven features obtained in Step 4 to train a random forest classifier integrating 50 decision trees. The input is the eleven features extracted from the phase sequences of different sign language signals, and the labels are the corresponding sign language signals. The actual meaning, such as "How old are you? (HOAR)", "Nice to meet you. (NTMY)" and so on. After training, the trained random forest classifier model is obtained.

In step 6, the trained random forest classifier model obtained in step 5 is obtained. For the new sign language data signal, its phase sequence features are extracted by the API of the RFID reader described in step 1, and after steps 1, 2, 3 and 4 are processed, eleven features for this new sign language data signal are obtained, Input it into the trained classifier described in step 5, and obtain the label corresponding to the new sign language data signal, that is, the actual meaning of the corresponding sign language. So far, sentence-level sign language recognition has been achieved.

The RFID-based sign language recognition method of this embodiment is specifically carried out according to the following steps:

Step 1: Collect the original sign language data signal

Commercial RFID devices based on COST RFID technology acquire the phase sequence of a specific target between the identification system and the passive tag by radio.

The RFID system consists of RFID tags, RFID readers and RFID directional antennas. Commercial RFID devices provide users with an application program interface. Through the API, users can obtain the received signal strength indication and passive tags to obtain the phase and other characteristics. API (Application Program Interface) is an application program interface, an application operation interface provided to users.

Among these eigenvalues, the phase sequence of the signal has high stability and is rarely disturbed by the environment, so the present invention selects the phase sequence as the feature of the present invention for sign language recognition.

The present invention designs 9 different sign languages according to the common communication language for deaf people in history museums, namely "Hi", "Da Tang Treasures Exhibition (DTTE)", "Digital Exhibition Hall (DEH)", "How old are you?(HOAY)","Tang Dynasty Mural Exhibition Hall(TDMEH)","Exhibition Exchange(EE)","Nice to meet you.(NTMY)","What is yourname?(WIYN)"," Exhibition Hall(EH)”, “Where is the toilet? (WITT)”, “Map”, “Where is the exit? (WITE)”, “Closed time (CT)”, “Cultural products (CP)” and “Parkinglocation” (PL)”. Each volunteer learned standard sign language before the experiment, and each volunteer performed each sign language gesture 100 times during the experiment, while the signal sampling rate was set to 270 times per second.

Step 2: De-noise the sign language signal

Perform phase offset correction and use threshold-based wavelet denoising method to remove the phase offset caused by environmental hardware and Gaussian noise, and obtain a pure phase sequence.

The phase sequence obtained from the RFID reader's API contains phase shifts due to hardware and Gaussian noise due to the environment.

Step 2.1: There are two types of phase shifts in the received signal, usually the phase sequence is shifted by π or 2π, as follows:

θ _true = θ-θ _i (θ _i =πor2π)

where i is the sequence number and θ is the phase. To calibrate the phase offset, the average of the resulting scent sequences was calculated and restored to the true phase sequence by adding or subtracting π or 2π, resulting in the phase sequence shown in Figure 4.

At the same time, in order to remove the Gaussian noise generated by the environment, the present invention continues to perform threshold-based discrete wavelet transform on the phase sequence obtained in the previous step to obtain the phase sequence shown in FIG. 5 .

Step 2.2: Obtain the sign language data signal from the API of the RFID reader through step 2.1. The phase sequence after noise removal is as follows:

θ={θ ₁ , θ ₂ ... θ _i ... θ _n }

where n is the number of phase sequences and θ(i) is the ith phase value.

At the same time, the same processing data is used to reduce the slight difference of the same gesture at different times. The calculation formula is as follows:

where S(i) is the normalized intermediate result, and a(i) is the ith phase sequence sample after normalization.

Step 3, using the signal processing method based on the standard deviation to realize the segmentation of the effective sign language data signal.

Step 3.1: Subtract the median and absolute value from each phase value in the sample:

m(i)=|a(i)-median(a)| where median(a) is the median and m(i) is the absolute value of each number in the phase sequence a(i) minus the median , the phase sequence diagram shown in Figure 6 is obtained.

Step 3.2: Calculate the standard deviation by grouping the data in sequence:

where d(i) is the standard deviation of the new series after calculating the standard, k is the length of the series, and μ is the mean series (k=50);

Step 3.3: For the standard deviation d(i) after grouping, a data group with a single data greater than the threshold r (r=0.1) is regarded as valid data and extracted.

After the above steps, the present invention extracts an effective sign language data signal and realizes the segmentation of the sign language data, as shown in FIG. 7 .

Step 4: By analyzing each feature between the same sign language signal and different sign language signals, select features that are stable in the same sign language signal but can be clearly distinguished between different sign language signals, and perform effective feature extraction.

4.1: Ten eigenvalues of the extracted valid data set: skewness, expected value, third-order center distance, mean, variance, standard deviation, kurtosis, energy, main frequency ratio to the maximum frequency peak;

4.2: Specifically introduce the extracted new feature SOS, that is, the Sum of Standard Deviations, which can measure the severity of data fluctuations.

SOS: The present invention assumes that n is the length of the array. Starting with the first set of arrays, each number and the following (k-1) numbers form an array of length k. The present invention obtains the standard deviation value of each array, then the present invention obtains (N-k+1) standard deviations, finds the sum, the sum of these (N-k+1) standard deviations, and obtains SOS:.

Function: You can compare the fluctuation of the softness of the data. When the data fluctuates, the value is large; when the data fluctuates less, it becomes small.

where N is the total number of data in the dataset, μ is the average of k data, k is the number of data per standard deviation, and k is taken as 5 in this experiment.

The fifth step is to use the random forest classifier and use 50 decision trees to realize sign language recognition.

Step 5.1: Disorganize the data and extract 1/10 of the data as the data test set, and the rest of the data as the training set, import the test set and the training set into the RF classifier for identification;

Step 5.2: Extract another 1/10 group of data as a data test set, and import the RF classifier cyclically until each group of data has been tested, and finally get the test result.

Example 4: Sign language recognition effect verification:

Experiment I:

The goal of Experiment I is to verify the effectiveness and recognition accuracy of the present invention. Firstly, 9 kinds of sign languages commonly used in historical museums are designed and used, and iterative calculation is performed for each sign language data signal, so that each sign language data signal can enter the training set and be evaluated in the test set.

Test results of Experiment I:

Demonstrate the accuracy of sign language recognition through confusion matrix. Each element in the confusion matrix represents the likelihood of one sign language being recognized as another. As shown in Figure 9, the average accuracy of sign language recognition reached 95.6%, among which the recognition rates of "DEH", "WITT" and "CP" reached 100%, while the recognition rates of "TDMEH" and "NTMY" were relatively The language is low, but overall it has a high recognition rate.

Experiment II:

The goal of experiment II is to verify the robustness of the RFID system of the present invention to the influence of surrounding dynamic multipaths against interference. The experimental scene of this experiment is based on the scene arrangement shown in Figure 8.

Compared with the interference of static multipath, dynamic multipath often has greater interference, so experiments are carried out in dynamic multipath scenarios. During the experiment, an additional volunteer walked around. The movement range was set as the center of the volunteers making gestures, and the radius was set to be 0.8-1.4m respectively (step length of 0.1m). During the test, the center volunteers kept making the same gestures.

Test results of experiment II:

As shown in Figure 10, when the range of the mover exceeds 1.4m, the interference to the system almost disappears, and it can be recognized at 1.2m. Therefore, the recognition range of other current methods is far from this accuracy. Dynamic multipath interference is robust.

Figure 11 shows the interferogram of the dynamic multipath for the sign language recognition system, and Figure 12 shows the relationship between the number of sign language recognition and the recognition accuracy.

Experiment III:

In Experiment III, the present invention compares the sign language recognition at sentence level of MyoSign and WiSign, the methods in other two sign language recognition literatures.

The test results of Experiment III are shown in Table 1.

Table 1 Comparison of sentence-level sign language recognition accuracy in different literatures

WiSign uses SVM (support vector machines), that is, support vector machines, to process sign language data signals, and MyoSign uses CNN (convolutional neural networks), that is, convolutional neural networks, to process sign language signals. It can be seen from Table 1 that the accuracy rate of WiSign is 93.8%, the accuracy rate of MyoSign is 93.1%, and the accuracy rate of the method of the present invention is 95.6%, which has certain advantages. Moreover, the WiSign method uses three Wi-Fi devices to improve performance, while the MyoSign method requires wearable devices and a large training dataset. In conclusion, the present invention has low cost and high identification.

In the above-mentioned embodiments, it may be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented in whole or in part in the form of a computer program product, the computer program product includes one or more computer instructions. When the computer program instructions are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer may be a general purpose computer, special purpose computer, computer network, or other programmable device. The computer instructions may be stored in or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be downloaded from a website site, computer, server, or data center Transmission to another website site, computer, server, or data center by wireline (eg, coaxial cable, fiber optic, digital subscriber line (DSL), or wireless (eg, infrared, wireless, microwave, etc.)). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, a data center, etc. that includes one or more available mediums integrated. The usable media may be magnetic media (eg, floppy disks, hard disks, magnetic tapes), optical media (eg, DVD), or semiconductor media (eg, Solid State Disk (SSD)), and the like.

The above are only specific embodiments of the present invention, but the protection scope of the present invention is not limited to this. Any person skilled in the art is within the technical scope disclosed by the present invention, and all within the spirit and principle of the present invention Any modifications, equivalent replacements and improvements made within the scope of the present invention should be included within the protection scope of the present invention.

Claims (10)

1. A sentence-level sign language recognition method is characterized by comprising the following steps:

step one, a commercial radio frequency identification device based on COST radio frequency identification technology obtains a phase sequence of a specific target between an identification system and a passive tag through radio to obtain an original sign language data signal;

step two, phase offset correction is carried out, and a wavelet denoising method based on a threshold value is utilized to remove phase offset caused by environmental hardware and Gaussian noise, so that a pure phase sequence is obtained;

thirdly, the effective sign language data signal is segmented by using a signal processing method based on the standard deviation;

step four, selecting and selecting the characteristics which are kept stable in the same sign language signal and can be obviously distinguished between different sign language signals through analyzing each characteristic between the same sign language signal and different sign language signals, and carrying out effective characteristic extraction;

and fifthly, recognizing the sign language by using a random forest classifier and 50 decision trees.

2. The sentence-level sign language identification method of claim 1, wherein in the second step, the phase offset correction and the removal of the phase offset caused by the environmental hardware and the gaussian noise by using the threshold-based wavelet denoising method to obtain the clean phase sequence comprise:

(1) the method for removing the offset caused by environmental hardware and Gaussian noise by using the wavelet denoising method based on the threshold value has the following formula:

θ_true＝θ-θ_n(θ_n＝πor 2π)；

wherein n is a serial number, θ_trueIs the true phase value after removing the phase offset, theta is the original phase sequence value, theta is the phase offset_nIs a phase offset value;

(2) the phase sequence after removing noise by obtaining the sign language data signal from the API of the RFID reader is as follows:

θ＝{θ₁，θ₂…θ_i…θ_n}；

where n is the number of phase sequences, θ is the phase sequence after noise removal, and θ is the number of phase sequences after noise removal_iIs a phase value in a phase sequence;

(3) the same processing data is used for reducing the slight difference of the same gesture at different moments, and the calculation formula is as follows:

where s (i) is the normalized intermediate result, θ (i) is the phase value in the phase sequence, min (θ) is the minimum value in the phase sequence, max (θ) is the maximum value in the phase sequence, and a (i) is the normalized ith phase sequence sample.

3. The sentence-level sign language identification method of claim 1, wherein in step three, the segmenting the valid sign language data signal by using the standard deviation-based signal processing method comprises:

(1) subtract the median from each phase value in the sample and find the absolute value:

m(i)＝|a(i)-median(a)|；

where mean (a) is the median, m (i) is the absolute value of each number in the phase sequence a (i) minus the median (a);

(2) and (3) grouping the data in sequence to calculate standard deviation:

wherein d (i) is the new sequence standard deviation after the calculation of the standard, k is the length of the sequence, μ is the average value sequence k-50;

(3) with the value of d (i), a data group whose individual data is greater than the threshold value r (r ═ 0.1) is valid data.

4. The sentence-level sign language identification method of claim 1, wherein in step four, the selecting of features that remain stable in the same sign language signal and can be distinguished clearly between different sign language signals by performing each feature analysis between the same sign language signal and different sign language signals, and performing effective feature extraction, comprises:

(1) extracting ten characteristic values of the obtained effective data group, wherein the ten characteristic values comprise skewness, an expected value, a third-order center distance, an average value, a variance, a standard deviation, kurtosis, energy and a main frequency ratio maximum frequency peak value;

(2) extracting the SOS value of the obtained valid data:

and (4) SOS: assuming N is the length of the array, starting with the first array, each number and the following (k-1) number make up an array of length k, get the standard deviation value for each array, get (N-k +1) standard deviations, find the sum of the (N-k +1) standard deviations, and get the SOS: (ii) a

Comparing the softness fluctuation of the data; when the data fluctuates, the value is large; when the data fluctuation is small, it becomes small, and the formula is as follows:

where N is the total number of data in the data set, μ is the average of k data, and k is the number of data per standard deviation.

5. The sentence-level sign language recognition method of claim 1, wherein in step five, the sign language recognition is implemented by using a random forest classifier and using 50 decision trees, and the method comprises the following steps:

(1) the data are scrambled, 1/10 groups of data are extracted to be used as a data test set, the rest data are used as a training set, and the test set and the training set are led into an RF classifier for identification;

(2) and extracting the data of other 1/10 groups as a data test set, and circularly importing the data into the RF classifier until each group of data is tested, thereby finally obtaining a test result.

6. A sentence-level sign language recognition system for implementing the sentence-level sign language recognition method according to any one of claims 1 to 5, wherein the sentence-level sign language recognition system comprises:

the sign language data signal acquisition module is used for acquiring a phase sequence of a specific target between an identification system and a passive tag through radio to obtain an original sign language data signal based on a commercial radio frequency identification device of COST radio frequency identification technology;

the signal denoising processing module is used for correcting phase offset and removing the phase offset caused by environmental hardware and Gaussian noise by using a wavelet denoising method based on a threshold value to obtain a pure phase sequence;

the sign language signal segmentation module is used for realizing the segmentation of effective sign language data signals by utilizing a signal processing method based on standard deviation;

the characteristic extraction module is used for selecting and selecting the characteristics which are kept stable in the same sign language signal and can be obviously distinguished between different sign language signals through analyzing each characteristic between the same sign language signal and different sign language signals, and carrying out effective characteristic extraction;

and the sign language recognition module is used for realizing sign language recognition by using 50 decision trees by utilizing a random forest classifier.

7. A computer device, characterized in that the computer device comprises a memory and a processor, the memory storing a computer program which, when executed by the processor, causes the processor to carry out the steps of:

the commercial radio frequency identification device based on COST radio frequency identification technology obtains a phase sequence of a specific target between an identification system and a passive tag through radio to obtain an original sign language data signal; performing phase offset correction and removing phase offset caused by environmental hardware and Gaussian noise by using a wavelet denoising method based on a threshold value to obtain a pure phase sequence; the method comprises the following steps of utilizing a signal processing method based on standard deviation to realize effective sign language data signal segmentation;

by analyzing each feature between the same sign language signal and different sign language signals, selecting and selecting the features which are stable in the same sign language signal and can be obviously distinguished between different sign language signals, and carrying out effective feature extraction; and (4) realizing sign language recognition by utilizing a random forest classifier and using 50 decision trees.

8. A computer-readable storage medium storing a computer program which, when executed by a processor, causes the processor to perform the steps of:

the commercial radio frequency identification device based on COST radio frequency identification technology obtains a phase sequence of a specific target between an identification system and a passive tag through radio to obtain an original sign language data signal; performing phase offset correction and removing phase offset caused by environmental hardware and Gaussian noise by using a wavelet denoising method based on a threshold value to obtain a pure phase sequence; the method comprises the following steps of utilizing a signal processing method based on standard deviation to realize effective sign language data signal segmentation;

by analyzing each feature between the same sign language signal and different sign language signals, selecting and selecting the features which are stable in the same sign language signal and can be obviously distinguished between different sign language signals, and carrying out effective feature extraction; and (4) realizing sign language recognition by utilizing a random forest classifier and using 50 decision trees.

9. An information data processing terminal characterized by being configured to implement the sentence-level sign language recognition system of claim 6.

10. An application of the sentence-level sign language recognition system of claim 6 in the technical field of artificial intelligence.

Images