CN104392237B - Fuzzy sign language identification method for data gloves - Google Patents

Abstract

The present invention discloses a fuzzy sign language recognition method for a data glove, comprising the following steps: obtaining hand motion data and performing fuzzy processing on the data to obtain a gesture frame sequence; performing recognition processing on the obtained gesture frame sequence according to a sign language database and a probability database to obtain a gesture frame sequence recognition result. The present invention effectively avoids the problem of low recognition rate caused by different palm sizes by performing fuzzy processing on the hand motion data, and by combining the sign language database and the probability database, the present invention can select the optimal recognition of the current gesture according to the previous and next gestures, thereby greatly improving the accuracy of gesture recognition. As a fuzzy sign language recognition method for a data glove, the present invention can be widely used in sign language recognition products.

Description

A Fuzzy Sign Language Recognition Method for Data Gloves

technical field

The invention relates to the field of gesture recognition, in particular to a fuzzy sign language recognition method for data gloves.

Background technique

Scientists at home and abroad have done a lot of research on gesture recognition. In 1994, Ramon M S and Dannil T developed a hand motion synthesis control and grasping system based on the physical constraints of the hand grasping process. In 1995, Lee Jintae and Kunii Tosiyasv L researched the motion image data of the hand obtained by the camera to automatically analyze the three-dimensional gestures and realize the reconstruction of the three-dimensional gestures. In 1997, the Glove Talk II system researched by Sidney S F of the University of Toronto in Canada is currently the most influential gesture interface system. He uses a neural network to convert user gestures into gesture language parameters, which are synthesized into language output through a speech synthesizer. my country's Gao Wen and others have also carried out research on sign language synthesis technology based on gesture and human behavior recognition.

At present, gesture recognition based on sensor data gloves directly obtains finger motion feature data from sensors, and then translates them into words or sounds that can be directly recognized by humans through matching algorithms. However, most of the current sign language translation technologies are limited to the recognition of words, and as the number of sign language databases increases, it is easy to cause confusion in translation, and due to the different sizes of hands, it is also easy to cause large differences in gesture recognition efficiency between different people. The accuracy rate is lower.

Contents of the invention

In order to solve the above technical problems, the object of the present invention is to provide a fuzzy sign language recognition method for a data glove that can improve the accuracy of gesture recognition.

The technical scheme adopted in the present invention is:

A fuzzy sign language recognition method for data gloves, comprising the following steps:

A. Obtain hand movement data and perform blurring processing on it to obtain a sequence of gesture frames;

B. Perform recognition processing on the obtained gesture frame sequence according to the sign language database and the probability database, and obtain the recognition result of the gesture frame sequence.

As a further improvement of the fuzzy sign language recognition method of a kind of data glove, described step A includes:

A1. Obtain the bending angle of each finger in the hand movement data, and obtain the corresponding bending state of each finger according to the preset bending membership function;

A2. Obtain the palm pitch angle in the hand movement data, and calculate it according to the preset pitch membership function, and obtain the largest result value as the corresponding pitch state;

A3. Obtain the palm inclination angle in the hand movement data, and calculate it according to the preset inclination membership function, and obtain the largest result value as the corresponding inclination state;

A4. Obtain the palm yaw angle in the hand movement data, and calculate it according to the preset yaw membership function, and the one with the largest result value is the corresponding yaw state;

A5. According to the calculated pitch state, tilt state and yaw state, combined with the preset rules, the corresponding palm orientation is obtained;

A6. According to the orientation of the palm and the bending state of each finger, a gesture frame is obtained, and then a sequence of gesture frames is obtained.

As a further improvement of the fuzzy sign language recognition method of the data glove, it is characterized in that: the step B includes:

B1. Obtain a sequence of gesture frames, and extract the gesture frames in sequence from beginning to end;

B2. Put the extracted gesture frames into the corresponding nodes in turn;

B3. Extract all the words corresponding to each gesture frame from the sign language database in turn, and attach them to the corresponding nodes until all the gesture frames on the gesture frame sequence complete the sign language database retrieval;

B4, the words attached to two adjacent nodes are combined in pairs according to the order of the nodes, and the words of the previous node point to the words of the next node in the combination;

B5. Indexing all combinations in the probability database to obtain the probability of each combination;

B6. Find out the sentence with the largest probability sum among the sentences formed by each combination, and obtain the gesture frame sequence recognition result.

As a further improvement of the fuzzy sign language recognition method of a data glove, the bending membership function in the step A1 is:

Among them, X∈U ₀ , U ₀ represents the bending angle of the finger, U ₀ = [0, 120], three fuzzy sets A ₀ of the bending angle of the finger are established on U ₀ to represent the state of the bending state as "straightening", A ₁ = indicates that the bending state is a "semi-bent" state, and A ₂ = indicates that the bending state is a "tight grip" state.

As a further improvement of the fuzzy sign language recognition method of a kind of data glove, the pitch membership function in the described step A2 is:

Among them, x∈U ₁ , U ₁ represents the pitch angle of the palm, U ₁ = [-90, 90], three fuzzy sets B ₀ of the pitch angle are established on U ₁ to represent the state that the pitch angle is "prostrate", B ₁ = indicates the state where the pitch angle is "horizontal", and B ₂ = indicates the state where the pitch angle is "upward".

As a further improvement of the fuzzy sign language recognition method of a kind of data glove, the oblique membership function in the step A3 is:

Among them, y∈U ₂ , U ₂ represents the inclination angle of the palm, U ₂ = [-180, 180], and the three fuzzy sets C ₀ establishing the inclination angle on U ₂ represent the state that the inclination angle is "left leaning", C ₁ = Indicates the state where the inclination angle is "horizontal", C ₂ = indicates the state where the inclination angle is "right-inclined", C ₃ = indicates the state where the inclination angle is "flip horizontal".

As a further improvement of the fuzzy sign language recognition method of a kind of data glove, the yaw membership function in the described step A3 is:

Among them, z∈U ₃ , U ₃ represents the yaw angle of the palm, U ₃ =[0, 360], and the three fuzzy sets D ₀ of the yaw angle established on U ₃ represent the state that the yaw angle is "front", D ₁ = indicates the state where the yaw angle is "right", D ₂ = indicates the state where the yaw angle is "back", D ₃ = indicates the state where the yaw angle is "left".

As a further improvement of the fuzzy sign language recognition method of a data glove, the sign language database uses gesture frames formed by finger bending state and palm orientation as indexes, and words corresponding to gestures as indexed content.

As a further improvement of the fuzzy sign language recognition method of a data glove, the probability of word combinations in the probability database is obtained by using the language model training tool SRILM.

The beneficial effects of the present invention are:

A fuzzy sign language recognition method for data gloves in the present invention effectively avoids the problem of low recognition rate caused by different palm sizes by performing fuzzy processing on the hand movement data, and by combining the sign language database and the probability database, the present invention can Select the optimal recognition of the current gesture according to the front and back gestures, which greatly improves the accuracy of gesture recognition.

Description of drawings

The specific embodiment of the present invention will be further described below in conjunction with accompanying drawing:

Fig. 1 is the step flow chart of the fuzzy sign language recognition method of a kind of data glove of the present invention;

Fig. 2 is a flow chart of step A of the fuzzy sign language recognition method of a data glove according to the present invention;

Fig. 3 is a step flow chart of step B of a fuzzy sign language recognition method of a data glove according to the present invention;

Fig. 4 is a schematic diagram of the composition of a gesture frame in a fuzzy sign language recognition method for a data glove according to the present invention.

detailed description

With reference to Fig. 1, the fuzzy sign language recognition method of a kind of data glove of the present invention comprises the following steps:

A. Obtain hand movement data and perform blurring processing on it to obtain a sequence of gesture frames;

B. Perform recognition processing on the obtained gesture frame sequence according to the sign language database and the probability database, and obtain the recognition result of the gesture frame sequence.

With reference to Fig. 2, as the further improvement of the fuzzy sign language recognition method of a kind of data glove, described step A comprises:

A1. Obtain the bending angle of each finger in the hand movement data, and obtain the corresponding bending state of each finger according to the preset bending membership function;

A2. Obtain the palm pitch angle in the hand movement data, and calculate it according to the preset pitch membership function, and obtain the largest result value as the corresponding pitch state;

A3. Obtain the palm inclination angle in the hand movement data, and calculate it according to the preset inclination membership function, and obtain the largest result value as the corresponding inclination state;

A4. Obtain the palm yaw angle in the hand movement data, and calculate it according to the preset yaw membership function, and the one with the largest result value is the corresponding yaw state;

A5. According to the calculated pitch state, tilt state and yaw state, combined with the preset rules, the corresponding palm orientation is obtained;

A6. According to the orientation of the palm and the bending state of each finger, a gesture frame is obtained, and then a sequence of gesture frames is obtained.

With reference to Fig. 3, as the further improvement of the fuzzy sign language recognition method of a kind of data glove, it is characterized in that: described step B comprises:

B1. Obtain a sequence of gesture frames, and extract the gesture frames in sequence from beginning to end;

B2. Put the extracted gesture frames into the corresponding nodes in turn;

B3. Extract all the words corresponding to each gesture frame from the sign language database in turn, and attach them to the corresponding nodes until all the gesture frames on the gesture frame sequence complete the sign language database retrieval;

B4, the words attached to two adjacent nodes are combined in pairs according to the order of the nodes, and the words of the previous node point to the words of the next node in the combination;

B5. Indexing all combinations in the probability database to obtain the probability of each combination;

B6. Find out the sentence with the largest probability sum among the sentences formed by each combination, and obtain the gesture frame sequence recognition result.

For example, gesture sequence S has two gesture frames A and B in turn, assuming gesture A has the words "you" and "that", and gesture B has the words "good" and "positive", assuming the probability of "hello" is 0.0052, the probability of "you are right" is "0.00045", the probability of "that's good" is 0.0078, and the probability of "that is right" is 0.00032, then the recognition result of gesture sequence S is the sentence with the highest probability, that is, the recognition result is "Well".

As a further improvement of the fuzzy sign language recognition method of a data glove, the bending membership function in the step A1 is:

Among them, X∈U ₀ , U ₀ represents the bending angle of the finger, U ₀ = [0, 120], three fuzzy sets A ₀ of the bending angle of the finger are established on U ₀ to represent the state of the bending state as "straightening", A ₁ = indicates that the bending state is a "semi-bent" state, and A ₂ = indicates that the bending state is a "tight grip" state.

As a further improvement of the fuzzy sign language recognition method of a kind of data glove, the pitch membership function in the described step A2 is:

Among them, x∈U ₁ , U ₁ represents the pitch angle of the palm, U ₁ = [-90, 90], three fuzzy sets B ₀ of the pitch angle are established on U ₁ to represent the state that the pitch angle is "prostrate", B ₁ = indicates the state where the pitch angle is "horizontal", and B ₂ = indicates the state where the pitch angle is "upward".

If the palm pitch angle is 42, substituting x=42 into the membership function calculation in Formula 2, it is obtained that B ₀ (42)=0.1, B ₁ (42)=0.3, B ₂ (42)=0, B ₁ > B ₀ >B ₂ , then the pitch angle of the x value input this time is horizontal.

As a further improvement of the fuzzy sign language recognition method of a kind of data glove, the oblique membership function in the step A3 is:

Among them, y∈U ₂ , U ₂ represents the inclination angle of the palm, U ₂ = [-180, 180], and the three fuzzy sets C ₀ establishing the inclination angle on U ₂ represent the state that the inclination angle is "left leaning", C ₁ = Indicates the state where the inclination angle is "horizontal", C ₂ = indicates the state where the inclination angle is "right-inclined", C ₃ = indicates the state where the inclination angle is "flip horizontal".

As a further improvement of the fuzzy sign language recognition method of a kind of data glove, the yaw membership function in the described step A3 is:

Among them, z∈U ₃ , U ₃ represents the yaw angle of the palm, U ₃ =[0, 360], and the three fuzzy sets D ₀ of the yaw angle established on U ₃ represent the state that the yaw angle is "front", D ₁ = indicates the state where the yaw angle is "right", D ₂ = indicates the state where the yaw angle is "back", D ₃ = indicates the state where the yaw angle is "left".

As a further improvement of the fuzzy sign language recognition method of a data glove, the sign language database uses gesture frames formed by finger bending state and palm orientation as indexes, and words corresponding to gestures as indexed content.

As a further improvement of the fuzzy sign language recognition method of a data glove, the probability of word combinations in the probability database is obtained by using the language model training tool SRILM.

Referring to FIG. 4 , a sequence of gesture frames in the present invention refers to a complete gesture sentence, including N gesture frames. A gesture frame is composed of 4 bytes (32 bits), of which the 32nd bit is reserved, the 26th to 30th and the 11th to 15th are the "palm facing" status of the left hand and the right hand respectively, and the 16th to 25th and The 1st to 10th digits store the bending state of the fingers of the left hand and the right hand respectively. One of the finger bending states occupies 2 digits. The thumb, index finger, middle finger, ring finger and little finger are sorted from high to low respectively, and the 0th digit is the checksum. bit.

The sign language frame sequence is arranged in chronological order by multiple sign language frames, and one sign language frame corresponds to multiple words or words, because a gesture represents different meanings in different contexts, and gestures have been analyzed in this method abstraction, so that some gestures with only slight differences are abstracted into the same sign language frame. The gesture recognition part is to extract the most suitable words from the sign language frames in the sign language frame sequence according to the statistical probability of the context and form the most ideal sentence with other sign language frames in the sign language frame sequence.

Sign language database and probability database must be established before sign language recognition.

The establishment of the sign language database refers to referring to the first and second volumes of "Chinese Sign Language", using the first part of the introduction method to propose the bending state of the fingers and the orientation state of the palm to form a sign language frame, and use this 4 bytes (32 bits) The sign language frame is used as the index, and the words corresponding to the gesture are used as the indexed content. The words corresponding to the same gesture are placed under an index. When searching with this index, the entire content of the index will be drawn. For example: the gestures of "you" and "that" are the same, then their indexes are the same, assuming that the index is A, when searching for A, "you" and "that" will be drawn.

The probability database refers to the frequency of occurrence of a single word in the corpus, and the collection of the frequency of each word followed by another word; the so-called corpus refers to daily sentences or articles in sign language. For a single word and the probability of a word followed by another word, it is obtained by using the language model training tool SRILM. and store them in the following format. The probability of the word or word combination is indexed by word or word combination. For example, the probability of "green leaf" is 0.000000147332, and the probability of "as for" and "appearance" is "0.000145517086". The index of the probability database is based on two words. If the combination of these two words cannot be found in \2-gram\, then find the individual probability of these two words in \1-gram\, set to P1 and P2, then the combined probability of the two words is P=P1*P2*e, where e is a natural constant of about 2.71828182845905.

The above is a specific description of the preferred implementation of the present invention, but the invention is not limited to the described embodiments, and those skilled in the art can also make various equivalent deformations or replacements without violating the spirit of the present invention. , these equivalent modifications or replacements are all within the scope defined by the claims of the present application.

Claims (7)

1. a fuzzy sign language recognition method of data gloves, is characterized in that, comprises the following steps: A. Obtain hand movement data and perform blurring processing on it to obtain a sequence of gesture frames; B. According to the sign language database and the probability database, the obtained gesture frame sequence is recognized and processed, and the gesture frame sequence recognition result is obtained; Described step A comprises: A1. Obtain the bending angle of each finger in the hand movement data, and obtain the corresponding bending state of each finger according to the preset bending membership function; A2. Obtain the palm pitch angle in the hand movement data, and calculate it according to the preset pitch membership function, and obtain the largest result value as the corresponding pitch state; A3. Obtain the palm inclination angle in the hand movement data, and calculate it according to the preset inclination membership function, and obtain the largest result value as the corresponding inclination state; A4. Obtain the palm yaw angle in the hand movement data, and calculate it according to the preset yaw membership function, and the one with the largest result value is the corresponding yaw state; A5. According to the calculated pitch state, tilt state and yaw state, combined with the preset rules, the corresponding palm orientation is obtained; A6. According to the palm orientation and the bending state of each finger, the gesture frame is obtained, and then the gesture frame sequence is obtained; The bending membership function in the step A1 is: <mrow> <msub> <mi>A</mi> <mn>0</mn> </msub> <mrow> <mo>(</mo> <mi>x</mi> <mo>)</mo> </mrow> <mo>=</mo> <mfenced open = "{" close = ""> <mtable> <mtr> <mtd> <mn>1</mn> <mo>,</mo> <mi>x</mi> <mo>&amp;le;</mo> <mn>35</mn> </mtd> </mtr> <mtr> <mtd> <mn>0</mn> <mo>,</mo> <mi>x</mi> <mo>&gt;</mo> <mn>35</mn> </mtd> </mtr> </mtable> </mfenced> </mrow> <mrow> <msub> <mi>A</mi> <mn>2</mn> </msub> <mrow> <mo>(</mo> <mi>x</mi> <mo>)</mo> </mrow> <mo>=</mo> <mfenced open = "{" close = ""> <mtable> <mtr> <mtd> <mn>1</mn> <mo>,</mo> <mi>x</mi> <mo>&amp;GreaterEqual;</mo> <mn>85</mn> </mtd> </mtr> <mtr> <mtd> <mn>0</mn> <mo>,</mo> <mi>x</mi> <mo>&lt;</mo> <mn>85</mn> </mtd> </mtr> </mtable> </mfenced> </mrow> Among them, x∈U ₀ , U ₀ represents the bending angle of the finger, U ₀ = [0, 120], three fuzzy sets of the bending angle of the finger are established on U ₀ , A ₀ represents the state of the bending state as "straightening", A ₁ indicates a state where the bending state is "semi-bent", and A ₂ indicates a state where the bending state is "grip". 2. the fuzzy sign language recognition method of a kind of data glove according to claim 1, is characterized in that: described step B comprises: B1. Obtain a sequence of gesture frames, and extract the gesture frames in sequence from beginning to end; B2. Put the extracted gesture frames into the corresponding nodes in turn; B3. Extract all the words corresponding to each gesture frame from the sign language database in turn, and attach them to the corresponding nodes until all the gesture frames on the gesture frame sequence complete the sign language database retrieval; B4, the words attached to two adjacent nodes are combined in pairs according to the order of the nodes, and the words of the previous node point to the words of the next node in the combination; B5. Indexing all combinations in the probability database to obtain the probability of each combination; B6. Find out the sentence with the largest probability sum among the sentences formed by each combination, and obtain the gesture frame sequence recognition result. 3. the fuzzy sign language recognition method of a kind of data glove according to claim 1, is characterized in that: the pitch membership degree function in the described step A2 is: <mrow> <msub> <mi>B</mi> <mn>0</mn> </msub> <mrow> <mo>(</mo> <mi>x</mi> <mo>)</mo> </mrow> <mo>=</mo> <mfenced open = "{" close = ""> <mtable> <mtr> <mtd> <mn>0</mn> <mo>,</mo> <mo>-</mo> <mn>90</mn> <mo>&amp;le;</mo> <mi>x</mi> <mo>&amp;le;</mo> <mn>40</mn> </mtd> </mtr> <mtr> <mtd> <mfrac> <mrow> <mi>x</mi> <mo>-</mo> <mn>40</mn> </mrow> <mn>20</mn> </mfrac> <mo>,</mo> <mn>40</mn> <mo>&lt;</mo> <mi>x</mi> <mo>&lt;</mo> <mn>60</mn> </mtd> </mtr> <mtr> <mtd> <mn>1</mn> <mo>,</mo> <mn>60</mn> <mo>&amp;le;</mo> <mi>x</mi> <mo>&amp;le;</mo> <mn>90</mn> </mtd> </mtr> </mtable> </mfenced> </mrow> <mrow> <msub> <mi>B</mi> <mn>2</mn> </msub> <mrow> <mo>(</mo> <mi>x</mi> <mo>)</mo> </mrow> <mo>=</mo> <mfenced open = "{" close = ""> <mtable> <mtr> <mtd> <mn>0</mn> <mo>,</mo> <mo>-</mo> <mn>40</mn> <mo>&amp;le;</mo> <mi>x</mi> <mo>&amp;le;</mo> <mn>90</mn> </mtd> </mtr> <mtr> <mtd> <mfrac> <mrow> <mo>-</mo> <mn>40</mn> <mo>-</mo> <mi>x</mi> </mrow> <mn>20</mn> </mfrac> <mo>,</mo> <mo>-</mo> <mn>60</mn> <mo>&lt;</mo> <mi>x</mi> <mo>&lt;</mo> <mo>-</mo> <mn>40</mn> </mtd> </mtr> <mtr> <mtd> <mn>1</mn> <mo>,</mo> <mo>-</mo> <mn>90</mn> <mo>&amp;le;</mo> <mi>x</mi> <mo>&amp;le;</mo> <mo>-</mo> <mn>60</mn> </mtd> </mtr> </mtable> </mfenced> </mrow> Among them, x∈U ₁ , U ₁ represents the pitch angle of the palm, U ₁ = [-90, 90], three fuzzy sets of pitch angles are established on U ₁ , B ₀ represents the state that the pitch angle is "prostrate", B ₁ means that the pitch angle is "horizontal", and B ₂ means that the pitch angle is "upward". 4. the fuzzy sign language recognition method of a kind of data glove according to claim 1, is characterized in that: the oblique membership degree function in the described step A3 is: Among them, y∈U ₂ , U ₂ represents the inclination angle of the palm, U ₂ = [-180, 180], three fuzzy sets of inclination angles are established on U ₂ , C ₀ represents the state of the inclination angle being "left-leaning", C ₁ indicates the state where the inclination angle is "horizontal", C ₂ indicates the state where the inclination angle is "right", and C ₃ indicates the state where the inclination angle is "flip horizontal". 5. the fuzzy sign language recognition method of a kind of data glove according to claim 1, is characterized in that: the yaw membership degree function in the described step A4 is: Among them, z ∈ U ₃ , U ₃ represents the palm yaw angle, U ₃ = [0, 360], three fuzzy sets of yaw angle are established on U ₃ , D ₀ represents the state that the yaw angle is "front" , D ₁ represents the state where the yaw angle is "right", D ₂ represents the state where the yaw angle is "back", and D ₃ represents the state where the yaw angle is "left". 6. The fuzzy sign language recognition method of a data glove according to claim 1, wherein the sign language database uses gesture frames formed by finger bending state and palm orientation as indexes, and words corresponding to gestures as indexed content. 7. The fuzzy sign language recognition method of a data glove according to claim 1, characterized in that: the probability of word combinations in the probability database is obtained by using a language model training tool SRILM.