CN104834922B - Gesture identification method based on hybrid neural networks - Google Patents

Abstract

The invention discloses a gesture recognition method based on a hybrid neural network. For a gesture image to be recognized and a gesture image training sample, firstly, a pulse-coupled neural network is used to detect and obtain noise points, and then a composite denoising algorithm is used to process the noise points, and then Cellular neural network is used to extract the edge points in the gesture image, and the connected areas are obtained according to the extracted edge points, and the fingertip detection is performed on each connected area by curvature to obtain the undetermined fingertip points, and the gesture area is obtained by excluding the interference of the face part, and then according to Gesture shape features are segmented, and Fourier descriptors that retain phase information are obtained according to the contour points of the segmented gesture area, and the first few Fourier descriptors are selected as gesture features; BP neural network is trained according to gesture features of gesture image training samples , input the gesture feature of the gesture image to be recognized into the BP neural network for recognition. The present invention improves the accuracy of gesture recognition through the application of various neural networks.

Description

Gesture Recognition Method Based on Hybrid Neural Network

technical field

The invention belongs to the technical field of gesture recognition, and more specifically relates to a gesture recognition method based on a hybrid neural network.

Background technique

With the rapid development of computer technology, human-computer interaction technology is becoming more and more popular in people's life. Human-computer interaction (Human-Computer Interaction, HCI) technology refers to an interactive process between a human and a computer that is performed by using a certain operation mode between the user and the computer. Its development has roughly gone through the pure manual operation stage, the language command control stage, the user interface stage, etc. However, with the continuous development of artificial intelligence and other technologies in recent years, it has gradually attracted attention to the development of human-computer interaction technology.

Now with the continuous expansion of computer applications, the existing human-computer interaction methods can no longer meet people's higher-level requirements for daily needs, and a more concise and friendly new human-computer interaction method is urgently needed. Since the ultimate goal of human-computer interaction is to achieve natural communication between humans and machines, in daily life, most of the information between humans is conveyed through body language or facial expressions, and only a small part is through natural communication. This shows that body language has a greater advantage in expressing human emotions or intentions. Since the hand plays an extremely important role in body language, the interaction method based on gesture behavior, that is, the gesture behavior recognition system, that is, the gesture recognition system has attracted widespread attention.

In general, a gesture recognition system is mainly composed of the following parts: gesture preprocessing, gesture segmentation, gesture modeling, gesture feature extraction, and gesture recognition. For the gesture preprocessing operation, it is mainly the denoising operation of the gesture image. At present, the common denoising algorithms include: mean filter, median filter, spatial low-pass filter, frequency domain low-pass filter, and pulse-coupled neural network. In the presence of such noise, the denoising ability of the current algorithm cannot achieve a good denoising effect, so designing a good denoising algorithm is very important for the later recognition process. For gesture segmentation operations, currently commonly used gesture segmentation methods include segmentation methods based on skin color information, segmentation methods based on motion information, and segmentation methods based on edge information. Because the segmentation method based on skin color information is easily disturbed by background information, and the segmentation method based on edge information cannot achieve a good segmentation effect, so how to design a good and effective segmentation algorithm is also crucial. For the gesture feature extraction operation, the feature extraction method based on the Fourier descriptor is currently the most widely used, but due to the rotation invariance of the method, the method has little change in the characteristics of the gesture after the gesture is rotated, so how to design a Fourier descriptors that are not rotation invariant are also crucial. For gesture recognition operations, currently common methods include template matching technology, support vector machines, neural network methods, hidden Markov models, etc. Therefore, how to choose a good gesture recognition method is also crucial for gesture recognition systems.

The neural network method refers to the science of simulating the human brain neurons by using some simple processing units, and connecting these simple processing units into a network in a certain way to realize the simulation of the human brain. Neural network methods often have the following advantages: parallel computing, distributed storage, robustness, nonlinear processing, and good adaptability and fault tolerance. Therefore, the neural network method can be applied in multiple scenarios. For example: gesture recognition, image segmentation, noise processing, etc.

At present, the neural network method has been more and more applied in the field of gesture behavior recognition. However, the application of neural network methods in the field of gesture recognition is limited to the stage of gesture recognition, and there are few applications in other stages of gesture recognition.

Contents of the invention

The purpose of the present invention is to overcome the deficiencies of the prior art, to provide a gesture recognition method based on a hybrid neural network, which uses a pulse-coupled neural network to improve the denoising effect of the gesture image, uses a cellular neural network to perform gesture segmentation, and adopts a method with rotational variability The Fourier descriptor is used as a gesture feature, and the BP neural network is used for gesture recognition, thereby improving the accuracy of gesture recognition

In order to achieve the above-mentioned purpose of the invention, the gesture recognition method based on the hybrid neural network of the present invention comprises the following steps:

S1: Extract the features of the gesture image to be recognized and the gesture image training samples, the specific steps include:

S1.1: The pulse-coupled neural network model of the gesture grayscale image is suggested. The gray value of each pixel of the current gesture grayscale image is used as the input of the corresponding neuron in the pulse-coupled neural network, and the pulse-coupled neural network is used to control the neural network. The pixel of the gesture image is detected. If the output state of the pixel is the ignition state, the element corresponding to the pixel in the detection result matrix is set to 1, otherwise it is set to 0; traverse each element of the detection result matrix, if the element If the value is 1, this element will be the center of the noise reduction processing window. The size of the noise reduction processing window is set according to the actual situation, and the values of other elements in the noise reduction processing window except the center point element are counted. If the element with a value of 0 If the number is greater than the preset threshold, it means that the center point is a noise point, otherwise it is not a noise point;

Calculate the two noise estimates H(i,j) and V(i,j) of the noise points according to the following formulas:

H(i,j)=|a(i,j)-b(i,j)|

Among them, a(i, j) is the gray value at the pixel point (i, j) in the image, and b(i, j) is the median output gray value of the pixel point after median filtering;

Among them, m ₁ (i, j) and m ₂ (i, j) respectively represent the gray values of the two points closest to the gray value of a (i, j) in the neighborhood where the pixel point (i, j) is located ;

If H(i,j)≥T ₁ , and V(i,j)≥T ₂ , use median filter to process the noise point, otherwise use mean value filter to process the noise point;

S1.2: performing histogram equalization on the gesture grayscale image denoised in step S1.1;

S1.3: Establish the cellular neural network model of the gesture grayscale image, and use the gray value of each pixel (i, j) of the equalized gesture grayscale image as the input u _ij of the corresponding cell in the cellular neural network model, according to The formula of the state transition process is iterated until the entire cellular neural network converges, and the output y _ij (t) of each cell is obtained; the output value of the cell corresponding to each pixel in the cellular neural network is traversed, when a certain pixel The output value is in the range of [0,1]. If the sum of the pixel values of other pixels in the corresponding neighborhood is greater than the preset threshold, the pixel is not an edge pixel, otherwise it is an edge pixel; when the output value is in [-1,0 ) range, not edge pixels;

S1.4: Obtain connected regions according to the edge pixels obtained in step S1.3, extract the contours of the connected regions, and perform fingertip detection on each connected region respectively. The fingertip detection method is:

Traverse each contour pixel point in the connected area, take this pixel point as a reference point, and mark the coordinates as p(p _x , p _y ,0), preset a distance constant L, and take the Lth point in front of point p along the contour direction point p ₁ (p _1x ,p _1y ,0), take the Lth point p ₂ (p _2x ,p _2y ,0) after point p, and calculate the vector with vector The cosine value cosα of the angle between them, if cosα is greater than the preset curvature threshold T, then it is determined that the point is a pending fingertip point, otherwise it is not regarded as a pending fingertip point;

Determine fingertip position vector product according to traversal direction The sign of , if the clockwise traversal of the overall outline of the gesture area is followed, the sign of the vector product should be negative, otherwise it is positive, and the pending fingertip point vector is calculated with vector vector product between If the sign of the vector product is the same as the sign corresponding to the fingertip position, it will be reserved as the pending fingertip point, otherwise it will not be reserved;

Determine whether the y-coordinate difference between the undetermined fingertip point with the largest y-coordinate and the undetermined fingertip point with the smallest y-coordinate among all undetermined fingertip points detected in the connected area exceeds half of the face height, and if so, the connected If the area is not a gesture area, otherwise it is regarded as a pending gesture area; further judge whether the number of pending fingertip points in each pending gesture area exceeds the preset number threshold, if yes, the connected area is a gesture area, otherwise not;

Find the main direction of the gesture area, and divide the gesture area according to the main direction according to the ratio of the length of the gesture to the width of 2 to obtain the divided gesture area;

S1.5: The gesture area obtained after step S1.4 is divided, the contour point coordinates of the gesture area are expressed in plural form, all the contour point coordinates are formed into a discrete sequence, and the number of contour points is n, and the discrete sequence is performed Fourier transform, get n Fourier coefficients z(k), k=0,1,...,n-1, calculate Fourier descriptor

where k'=1,2,...,n-1, Indicates the angle between the main direction of the gesture area and the x-axis.

Select the first Q constituent feature vectors in the Fourier descriptor;

S2: The feature vector of the gesture image of the training sample is input into the BP neural network as a training sample, and the corresponding gesture image category is used as the output of the BP neural network to train the BP neural network;

S3: Input the feature vector of the gesture image to be recognized into the BP neural network trained in step S2, and output the recognized gesture image category.

The gesture recognition method based on the hybrid neural network of the present invention, for the gesture image to be recognized and the gesture image training samples, first uses the pulse-coupled neural network to distinguish and detect the noise point and the edge point, and then uses the composite denoising algorithm to process the noise point, Then the cellular neural network is used to extract the edge points in the gesture image, and the connected areas are obtained according to the extracted edge points, and the fingertip detection is performed on each connected area by the curvature to obtain the undetermined fingertip points, and then the interference of the face part is excluded to obtain the gesture region, and then segmented according to the characteristics of the hand shape to obtain the segmented gesture region; obtain the Fourier descriptors that retain the phase information according to the contour points of the gesture region, and select the first few Fourier descriptors as gesture features; according to the gesture image The gesture features of the training samples are used to train the BP neural network, and the gesture features of the gesture images to be recognized are input into the BP neural network for recognition.

The present invention has the following beneficial effects:

(1) Use the pulse-coupled neural network to distinguish noise points and edge points, and combine the composite denoising algorithm to denoise the gesture image, which can improve the denoising effect;

(2) Gesture segmentation combines the coarse segmentation of cellular neural network and the fine segmentation based on gesture shape features, which can improve the accuracy of gesture segmentation;

(3) The Fourier descriptor is used for the gesture feature, which retains the phase information and can improve the recognition rate.

Description of drawings

Fig. 1 is the flowchart of the gesture recognition method based on hybrid neural network of the present invention

Fig. 2 is the flowchart of gesture image feature extraction in the present invention

Figure 3 is a flow chart of gesture fine segmentation combined with gesture shape characteristics

Fig. 4 is the schematic diagram of fingertip detection of the present invention;

Fig. 5 is an example diagram of gesture coarse segmentation;

FIG. 6 is an example diagram of fine segmentation of gestures.

detailed description

Specific embodiments of the present invention will be described below in conjunction with the accompanying drawings, so that those skilled in the art can better understand the present invention. It should be noted that in the following description, when detailed descriptions of known functions and designs may dilute the main content of the present invention, these descriptions will be omitted here.

Example

Fig. 1 is a flow chart of the gesture recognition method based on the hybrid neural network of the present invention. As shown in Figure 1, the gesture recognition method based on hybrid neural network of the present invention comprises the following steps:

S101: Extracting features of samples to be identified and training samples:

First, feature extraction needs to be performed on gesture images to be recognized and gesture image training samples. Fig. 2 is a flowchart of gesture image feature extraction in the present invention. As shown in Figure 2, gesture image feature extraction in the present invention comprises the following steps:

S201: gesture image denoising preprocessing:

The present invention uses a denoising algorithm based on a combination of PCNN-Pulse Coupled Neural Network (PCNN-Pulse Coupled Neural Network) and a composite denoising algorithm to denoise gesture grayscale images. The edge points are distinguished and detected, and then the compound denoising algorithm is used to perform denoising operations according to the type of noise points, so as to achieve the purpose of removing various noises under the premise of retaining edge information.

Each neuron of a pulse-coupled neural network consists of three parts: a receiving part, a modulating part and a pulse generator. Pulse-coupled neural network is a common method for image noise reduction preprocessing, and its main function is to remove salt and pepper noise. When the pulse-coupled neural network is used in the field of image noise reduction, it can be understood as a two-dimensional single-layer local connection network. In this network, neurons correspond to pixels in the grayscale image to be processed one-to-one, and are relatively Neighboring neurons are also interconnected. In the process of denoising, the gray value of each pixel of the image to be processed can be understood as the feedback input of neurons, while the output of each neuron is only used as the input of adjacent neurons, and the output of each neuron There are only two output states: ignition state and non-ignition state, which can be recorded as 1 and 0 respectively. Since the pixel corresponding to the noise is quite different from the surrounding pixels, the noise point can be judged by using the emission characteristics of the pulse-coupled neural network combined with the characteristics of the noise itself. The specific judgment method is as follows:

Establish the pulse-coupled neural network model of the gesture gray-scale image; use the gray value of each pixel of the current gesture gray-scale image as the input of the corresponding neuron in the pulse-coupled neural network, and then use the pulse-coupled neural network’s firing characteristics to analyze the entire image. Pixels are detected. If the output state of the pixel is the ignition state, the element corresponding to the pixel in the detection result matrix is set to 1, otherwise it is set to 0. It can be seen that the size of the detection result matrix is the same as that of the image to be processed; Noise processing window size, in the present embodiment, be 3 * 3; Traverse each element of detection result matrix, if element value is 1, just be ignition state, then take this element as the center of noise reduction processing window, statistical noise reduction Process the values of other elements in the window except the center point element (that is, the detection results of other pixels other than the pixel corresponding to the center point), if the number of elements with a value of 0 (that is, the non-ignition state) is greater than the preset threshold, it means that the center The center point is a noise point, and in other cases, the center point is not a noise point. So as to achieve the purpose of judging and distinguishing noise points and edge points. The number threshold is generally half the number of elements in the denoising processing window.

After the noise point is judged, the composite denoising algorithm is used to perform the corresponding denoising operation. The main method is:

Assume that a(i,j) is the grayscale value at the pixel point (i,j) in the image, 1≤i≤M, 1≤j≤N, M represents the number of pixels in each row of the gesture grayscale image (ie column number), N represents the number of pixels (that is, the number of rows) in each column of the gesture grayscale image, and b(i, j) is the median output gray value of the pixel after median filtering. In order to achieve the purpose of Gaussian noise, the difference between the pixel value of the noise point and the median output gray value is used as the noise estimation value, as shown in the following formula (1)

H(i,j)=|a(i,j)-b(i,j)| (1)

Due to the different types of noise, if the above estimation method is simply used, the purpose of distinguishing multiple noises cannot be achieved. Therefore, on the basis of the above formula, another noise estimation value V(i,j) is introduced. The parameter is The average value of the gradient sum of the pixel value a(i,j) at the pixel point (i,j) and the two adjacent points m ₁ (i,j) and m ₂ (i,j), as shown in the following formula (3) shown

Among them, m ₁ (i, j) and m ₂ (i, j) respectively represent the gray values of the two points closest to the gray value of a (i, j) in the neighborhood where the pixel point (i, j) is located .

If the thresholds are set to T ₁ and T ₂ , the corresponding processing of different noises can be realized through the relationship between the above two noise estimates and the thresholds. The specific method is as follows:

If H(i,j)≥T ₁ , and V(i,j)≥T ₂ , then it is determined that the type of noise point is salt and pepper noise or impulse noise, and the noise point is processed by median filtering, that is, the noise point The gray value of the point is modified to the output value of the median filter. If H(i,j)<T ₁ , or H(i,j)≥T ₁ and V(i,j)<T ₂ , the noise type is determined For Gaussian noise, the noise point is processed by mean filtering, that is, the gray value of the noise point is changed to the output value of mean filtering.

_In the above algorithm, the selection _of thresholds T1 and T2 is very important to the results of the composite denoising algorithm. Among them, the commonly used threshold selection method is the mean absolute deviation algorithm (MAD algorithm). According to the algorithm, it can be known that T ₁ =3.5δ _ij , where δ _ij represents the average absolute dispersion of all pixels within the denoising window of the pixel (i,j). The selection of the threshold T ₂ is mainly aimed at the texture that may appear in the gesture image. According to the MAD algorithm and experimental experience, the value of T ₂ is usually selected as an integer of 6-10.

S202: Histogram equalization:

Histogram equalization refers to the method of using the image histogram to adjust the contrast of the image, so that the gray histogram of the original image is changed from a relatively concentrated gray area to a uniform distribution in the global range. In the present invention, the histogram equalization process is performed on the gesture grayscale image after the denoising process in step S201, in order to enlarge the difference in the grayscale value of the foreground and background parts of the gesture image. Histogram equalization is a commonly used image contrast enhancement method at present, and its specific steps will not be repeated here.

S203: Coarse gesture segmentation based on cellular neural network:

Like the pulse-coupled neural network, the neurons in the cellular neural network are in one-to-one correspondence with the pixels in the gesture grayscale image, and the cell in row i and column j is C(i,j) (corresponding to the gesture grayscale image pixel (i,j)), cell C(i,j) consists of four parts: input variable u _ij , state transition variable x _ij , output variable yi _j and threshold I. The cells of the cellular neural network are locally interconnected, and the cell C(i,j) is only connected to the cells in its neighborhood N _r (i,j), but has no direct connection relationship with other cells. Cell C(i,j) neighborhood N _r (i,j) can be defined as:

N _r (i,j)=C(k,l)|max(ki,lj)≤r (3)

Among them, r is a positive integer, 1≤i, k≤M, 1≤j, l≤N, M represents the number of pixels in each row of the gesture grayscale image, and N represents the number of pixels in each column of the gesture grayscale image. That is, the neighborhood of cell C(i,j) is the range included by a square with side length 2r+1 centered on C(i,j).

The main formula of CNN is:

State transition process:

Output equation:

Among them, 1≤i, k≤M, 1≤j, l≤N; t represents the number of iterations; A(k,l) represents the neighborhood N _r (i,j) where the cell C(i,j) is located The feedback weight of the cell C(k,l); B(k,l) represents the control of the cell C(k,l) in the neighborhood N _r (i,j) where the cell C(i,j) is located Weight, that is, other elements in template B except the central position element. Here the value of (k,l) depends on the definition of the neighborhood N _r (i,j).

The feedback template A and the control module B are both (2r+1)×(2r+1) matrices, I represents the threshold template of the cellular neural network, and the values of A, B and I comprehensively determine the input quantity u of the cellular neural network The corresponding relationship between _ij , output quantity yi _j and state transition quantity x _ij . Therefore, for the cellular neural network model, how to correctly design the feedback template A, the control template B and the value of the threshold I is very important.

The template design method that the present invention adopts is the template design method based on the combination of algebraic knot and predecessor's template design experience, and the format of template A, B, I is generally designed as follows:

I=-d (8)

Among them, a, b, c, d are all normal numbers.

Establish the cellular neural network model of the gesture grayscale image, and use the gray value of each pixel (i, j) of the gesture grayscale image after equalization as the input u _ij of the corresponding cell in the cellular neural network model, according to the formula of the state transition process Iterate until the entire cellular neural network converges, and each cell has an output y _ij (t). According to the output equation, the output value y _ij (t) of the cellular neural network is between 1 and -1, when y _ij (t) is 1, it represents all black; when y _ij (t) is -1, Represents all white.

The basic principle of judging whether a pixel is an edge point is: when a pixel value is completely black, that is, +1, if the sum of the pixel values in its corresponding neighborhood is greater than the set threshold parameter, then the pixel is not At this time, the pixel value tends to be completely white; on the contrary, if the sum of the pixel values in the corresponding neighborhood is less than the set threshold parameter, this pixel represents an edge pixel, and the pixel value at this time tends to be completely black. When the value of this pixel is all white, that is, -1, the value of this pixel will tend to be all white regardless of the value of each pixel in its corresponding neighborhood.

According to the above principles, the method for judging whether a certain pixel is an edge point in the present invention is: traverse the output value of the cell corresponding to each pixel in the cellular neural network, when the output value of a certain pixel is in [0,1] Within the range, if the sum of the pixel values of other pixels in the corresponding neighborhood is greater than the preset threshold, the pixel is not an edge pixel, otherwise it is an edge pixel; when the output value is in the range of [-1,0), it is not an edge pixel . The threshold of the neighborhood pixel value sum is set according to the actual situation.

S204: Combining gesture shape features to perform gesture segmentation:

Fig. 3 is a flow chart of fine segmentation of gestures combined with gesture shape features. As shown in Figure 3, the gesture subdivision of the present invention includes the following steps:

S301: Extract connected regions and contours:

According to the edge pixels obtained by using the cellular neural network, the connected area is calculated, so as to remove the interference of other background information, and only keep the human hand and face area. In this embodiment, the algorithm used to obtain the connected regions is the two_pass algorithm. Then extract the contour of the connected region. This embodiment uses the search mark method to extract the contour. The specific process is: systematically scan the image after the connected region is extracted above. If a point in the connected region is encountered, use this point as the starting point, then trace its edge, and mark the pixels above the edge. When the scanned contour reaches complete closure, return to the previous position and continue scanning until new pixel information is found. Extracting connected regions and contours can also use other methods as needed.

S302: Perform fingertip detection on each connected region:

Fingertip detection is performed on each of the obtained connected regions to determine whether it is a gesture region. In general, when gesture recognition is performed, fingers are separated, so fingertip detection can be performed through curvature calculation. Fig. 4 is a schematic diagram of fingertip detection in the present invention. As shown in 4, the method of fingertip detection is:

Traverse each contour pixel point in the connected area, and use this pixel point as a reference point, the coordinates are marked as p(p _x , p _y ,0), (p _x , p _y ) means the position of the reference point in the gesture image Two-dimensional coordinates, preset a distance constant L, take the L-th point p ₁ (p _1x ,p _1y ,0) in front of point p along the contour direction, then point p and point p ₁ form a straight line, and then follow the contour direction Take the L-th point p ₂ (p _2x ,p _2y ,0) behind point p, then point p and point p ₂ can also form a straight line, and an angle will be formed between the two, and the angle is recorded as α; convert the vector with vector The cosine value of the included angle is used as the curvature result to be calculated, that is, the curvature calculation formula is:

If cosα is greater than the preset curvature threshold T, it is determined that the point is a pending fingertip point. The size of the threshold T is set according to the distance constant L. When the distance constant L is larger, the threshold T is also larger. Usually, the distance constant L cannot be too small or too large, and it is generally set according to 1/4 to 1/2 of the average finger length.

For the interference of the groove part of the finger, the vector with vector Determine the sign of the vector product between . It can be seen from Figure 4 that when the point p is located at the fingertip The sign of the vector product is the same as when the point p is in the groove position The sign of the vector product is different, so it can be obtained by to determine the position of point p. It is for this purpose that the coordinates of the points p, p ₁ and p ₂ are expressed in three-dimensional Cartesian coordinates. fingertip position The sign of the vector product is related to the traversal direction. When traversing the overall outline of the gesture area clockwise, according to the right-hand rule of the vector product, the position of the fingertip The vector product is perpendicular to the image inward, which is negative. When traversing counterclockwise according to the overall outline of the gesture area (traversal direction as shown in Figure 4), the fingertip position The vector product is positive if it is perpendicular to the image outwards. according to fingertip position The sign of the vector product, thus removing the interference of the groove part. That is, to judge the pending fingertip If the sign of the vector product is the same as that corresponding to the fingertip position, it is reserved as the pending fingertip point, otherwise it is not reserved.

S303: Determine the gesture area:

After the fingertip point is detected, it is also necessary to judge the fingertip point, so as to remove the interference caused by some parts of the face with a curvature greater than the threshold due to the angle problem, and determine the gesture area. The present invention has adopted double judgment method:

First judge whether the y-coordinate difference between the undetermined fingertip point with the largest y-coordinate detected in the connected area and the undetermined fingertip point with the smallest y-coordinate detected exceeds half of the face height, and if so, the connected area Not a gesture area, otherwise it is a pending gesture area. The reason why the distance is set to half the height of the face here is obtained through experimental testing, so that the interference of the face part can be removed under the premise of completely retaining the correct fingertip points.

It is further judged whether the number of undetermined fingertip points in each undetermined gesture area exceeds the preset number threshold, if yes, the connected area is a gesture area, otherwise it is not. The number of fingertip points obtained in the actual gesture area is related to the curvature threshold T. Therefore, in practical applications, the threshold of the number of fingertip points can be obtained by statistically analyzing the experimental results of several gesture training samples.

S304: Gesture area segmentation:

Through the above operations, the interference of other connected areas such as faces is removed, and the gesture area is obtained. However, the gesture area may not only include the palm of the person, but may also include the wrist and other parts. In general, the effective information of human gestures is concentrated in the palm of the person, and the information of the wrist and other parts can basically be ignored. Therefore, in order to make the later feature extraction and tracking efficient and effective, it is necessary to segment the gesture area to achieve the purpose of retaining only the fingers and palms.

According to the shape characteristics of the human hand, the present invention realizes the segmentation of the gesture according to the ratio of the length of the gesture to the width of the gesture being approximately equal to 2. Before performing segmentation, it is necessary to know the main direction of the gesture area. In this embodiment, the method for obtaining the main direction of the gesture is: obtain the centroid of the gesture area, and then obtain the vectors from the centroid to each fingertip point, and carry out these vectors On average, the direction of the average vector is the main direction of the gesture area. Then the gesture is segmented according to the main direction of the gesture area. The segmentation method adopted in this embodiment is: according to the main direction of the gesture area, the circumscribed rectangle of the gesture area is obtained, the side parallel to the main direction is long, and the side perpendicular to the main direction is wide, and the wide side where the fingertip point is located is selected. Starting from the broad side, intercept the circumscribed rectangle along the long side with a distance of twice the length of the wide side. The gesture area contained in the circumscribed rectangle is the gesture area to be segmented and only the fingers and palms are reserved.

S205: Extracting gesture features using a Fourier descriptor that retains phase information:

For the gesture area segmented in step S204, the present invention designs a Fourier descriptor that retains phase information to extract gesture feature information, thereby removing the rotation invariance of traditional Fourier descriptors and achieving the purpose of distinguishing rotation gestures.

The discrete Fourier coefficient z(k) can be expressed as:

Among them, p(i) represents the i-th data in the discrete sequence, n represents the number of data in the discrete sequence, e represents the natural constant, and j is the imaginary unit. In the present invention, since it is the gesture contour that needs to be transformed, the discrete sequence p(i) is the complex form of the coordinates of the gesture region contour pixels obtained by segmentation in step S104.

The inverse Fourier transform can be expressed as:

According to the basic property z(k)=z ^* (nk) of Fourier transform, the high-frequency part from K+1 to nK-1 in the Fourier transform form z is removed, wherein, z* here represents the total of z The complex form of the yoke; the value range of K is: [0,n/2]. Then inverse Fourier transform is performed on z that removes the high-frequency part, and a curve similar to the original Fourier transform will be obtained, but the curve becomes smoother, and this curve becomes the Kth approximate curve of the original Fourier change curve . Wherein, the subset {z(k)nK<k≤K} of the Fourier coefficients described above is the Fourier descriptor used to extract gesture features.

The Fourier descriptor has a certain relationship with the scale, direction and starting position of the curve. Therefore, in order to ensure that the recognition algorithm has rotation, translation and scale invariance, the Fourier descriptor needs to be normalized. According to the basic properties of the Fourier transformation, it can be proved that when the Fourier coefficient is used to represent the contour, the coefficient amplitude ||z(k)|| has rotation invariance, translation invariance and independence of the starting point position, where 0≤k ≤n-1, and since Z[0] does not have translation invariance, the value range of k is set to [1,n-1]. In order to achieve the scale invariance of the Fourier descriptor, the amplitude Z(k)|| of each coefficient except Z[0] can be divided by ||Z(1)|| to achieve scale invariance characteristic. The Fourier descriptor S[k′] after the normalization operation can be expressed as:

Among them, 1≤k′≤n-1; || || represents the modulo operator.

For a detailed description of the normalized Fourier descriptor, please refer to the document "Song Ruihua. Gesture Recognition Algorithm Based on Fourier Descriptor [D]. Xidian University, 2008".

In order to remove the rotation invariance of the traditional Fourier descriptor, the present invention retains the phase information after rotation, and improves the normalized form of the Fourier descriptor It can be expressed as:

in, Indicates the angle between the main direction of the gesture area and the x-axis, and j is the imaginary unit. The above Fourier descriptor S[k'] retains the phase information of gesture rotation, so this descriptor does not have rotation invariance. Therefore, the present invention adopts the Besides The coefficient of is used as the feature of the gesture area. This feature has translation and scale invariance, and has nothing to do with the initial position of the gesture contour curve, and at the same time has rotation variability. This feature vector can achieve the purpose of distinguishing rotation gestures. Since the number of contour points in different gesture regions is not necessarily the same, in practical applications, only the first Q constituent feature vectors are uniformly selected in the Fourier descriptor, and the size of Q can be determined according to the actual situation.

S102: Train the BP neural network according to the training samples:

The feature vector of the gesture image of the training sample is input into the BP neural network as a training sample, and the corresponding gesture image category is used as the output of the BP neural network to train the BP neural network. The BP neural network is a commonly used neural network. The specific composition, parameters and training methods of the network will not be repeated here.

S103: Perform gesture recognition on samples to be recognized:

Input the feature vector of the gesture image to be recognized into the BP neural network trained in step S102, and output the recognized gesture image category.

In order to illustrate the technical effects of the present invention, the present invention has been verified experimentally. The selected gesture training samples are divided into four parts: gestures facing up, gestures facing down, gestures facing left, and gestures facing right. The number of training samples in each part is 80. Similarly, test samples are selected from these four types of images. Sample size 40. For the convenience of demonstration, here we only select the sample with the gesture facing up to illustrate the implementation process. The size of each picture in the sample is 256×256, and the gray level is 256.

First, image denoising needs to be performed on the upward samples. Since the scale size of the sample picture is 256×256, and when the pulse-coupled neural network is used in the field of image noise reduction, the number of neurons corresponds to the pixel of the image to be processed, so the number of neurons in the pulse-coupled neural network The number is set to 65536, and the parameters of the pulse-coupled neural network model adopted in this embodiment are set to: the number of neuron iterations τ=10, the neuron connection strength β=3, the dynamic threshold parameter _θij =1, and the amplification factor of the threshold output V _θ = 20, the attenuation coefficient a _θ = 0.2 of the threshold function, and then use the emission characteristics to detect the pulse-coupled neural network, and then judge the noise points through the detection results, and then use the composite denoising algorithm to remove the noise points according to the type of noise points Noise operation, wherein the parameters of the composite denoising algorithm are set to T ₁ =3.5δ _ij , where S _k represents the noise window, the size of the noise window is consistent with the detection window size of the pulse-coupled neural network, the size is 3×3, T ₂ = 8.

After performing histogram equalization on the denoised gesture image, the edge of the gesture in the gesture image is detected by using the cellular neural network, and the rough segmentation of the gesture image is realized. In this embodiment, the neighborhood of each cell in the cellular neural network The size of is 3*3, and the template used is:

Fig. 5 is an example diagram of gesture coarse segmentation.

Then the gesture image is finely segmented by combining gesture shape features. The size of the constant L is 80, and the size of the threshold T for curvature calculation is 0.5. FIG. 6 is an example diagram of fine segmentation of gestures. It can be seen that after fine segmentation of gestures, the influence of areas such as faces can be eliminated, and more accurate gesture areas can be obtained.

Then the contour point coordinates of the gesture area obtained by subdivision are constructed into a discrete sequence, and the Fourier coefficients are obtained after Fourier transform, and then normalized according to formula (13), and the normalized Fourier description Select the first 200 constituent gesture feature vectors in the subsection.

The gesture feature vector of the training sample is used to train the BP neural network, wherein the number of the input layer of the BP neural network is determined by the gesture feature vector, and the number of the output layer is determined by the gesture sample type, and the number of the input layer adopted in the present invention is 200, the number of hidden layers is 10, and the number of output layers is 4. The output result can be expressed in binary form 0001, 0010, 0100, 1000, where 0001 means the gesture is facing up, 0010 means the gesture is facing down, 0100 means the gesture is facing the left, and 1000 means the gesture is facing the right. According to the result of the gesture output, determine what kind of gesture it belongs to Types of.

In order to verify the quality of the noise reduction effect of the new denoising algorithm based on the combination of pulse-coupled neural network and composite denoising algorithm designed by the present invention, the denoising algorithm designed in the present invention is combined with simple composite denoising algorithm and median filtering For comparative analysis, the main index of comparison is the peak signal-to-noise ratio (PSNR). Table 1 is a comparison table of PSNR between the denoising algorithm of the present invention and the comparison algorithm.

Table 1

It can be seen from Table 1 that in the case of the same noise density, the PSNR value of the denoising method proposed by the present invention is obviously higher than that of the median filter and the simple compound denoising algorithm. It can be seen that the denoising algorithm combined with the pulse-coupled neural network and the compound denoising algorithm designed by the present invention has a good denoising effect.

In addition, the recognition effect of the traditional Fourier descriptor is also used for comparison, and the comparison index is the recognition rate of gesture samples. Table 2 is a statistical table of gesture sample recognition results of traditional Fourier descriptors. Table 3 is a statistical table of gesture sample recognition results of the Fourier descriptor of the present invention.

Table 2

table 3

By comparing the results of Table 2 and Table 3, it can be seen that the traditional Fourier descriptor cannot recognize the gesture with a large rotation, the recognition rate is only about 71%, and the recognition rate is low, so this method is used for the rotation gesture. It doesn't work very well for scenes with different meanings. The improved Fourier descriptor of the present invention can tolerate a certain angle of gesture rotation. Although it will be considered as two different images when the angle is too large, it is verified by experiments that the present invention still achieves a recognition rate of about 91%. A good gesture recognition effect has been achieved.

Although the illustrative specific embodiments of the present invention have been described above, so that those skilled in the art can understand the present invention, it should be clear that the present invention is not limited to the scope of the specific embodiments. For those of ordinary skill in the art, As long as various changes are within the spirit and scope of the present invention defined and determined by the appended claims, these changes are obvious, and all inventions and creations using the concept of the present invention are included in the protection list.

Claims (4)

1. A gesture recognition method based on a hybrid neural network is characterized by comprising the following steps:

s1: the method comprises the following steps of extracting the characteristics of a gesture image to be recognized and a gesture image training sample, and specifically comprises the following steps:

s1.1: suggesting a pulse coupling neural network model of the gesture gray image, taking the gray value of each pixel point of the current gesture gray image as the input of a corresponding neuron in the pulse coupling neural network, detecting the pixel point of the gesture image by using the issuing characteristic of the pulse coupling neural network, setting an element corresponding to the pixel point in a detection result matrix to be 1 if the output state of the pixel point is an ignition state, and otherwise setting the element to be 0; traversing each element of the detection result matrix, if the element value is 1, taking the element as the center of a noise reduction processing window, setting the size of the noise reduction processing window according to the actual situation, counting the values of other elements except the central point element in the noise reduction processing window, if the number of the elements with the value of 0 is greater than a preset threshold value, indicating that the central point is a noise point, and if the number of the elements with the value of 0 is not the noise point;

two kinds of noise estimation values H (i, j) and V (i, j) of the noise point are calculated by the following formulas, respectively:

H(i,j)＝|a(i,j)-b(i,j)|

wherein a (i, j) is the gray value of a pixel point (i, j) in the image, and b (i, j) is the median output gray value of the pixel point after median filtering;

<mrow> <mi>V</mi> <mrow> <mo>(</mo> <mi>i</mi> <mo>,</mo> <mi>j</mi> <mo>)</mo> </mrow> <mo>=</mo> <mfrac> <mrow> <mo>|</mo> <msub> <mi>m</mi> <mn>1</mn> </msub> <mrow> <mo>(</mo> <mi>i</mi> <mo>,</mo> <mi>j</mi> <mo>)</mo> </mrow> <mo>-</mo> <mi>a</mi> <mrow> <mo>(</mo> <mi>i</mi> <mo>,</mo> <mi>j</mi> <mo>)</mo> </mrow> <mo>|</mo> <mo>+</mo> <mo>|</mo> <msub> <mi>m</mi> <mn>2</mn> </msub> <mrow> <mo>(</mo> <mi>i</mi> <mo>,</mo> <mi>j</mi> <mo>)</mo> </mrow> <mo>-</mo> <mi>a</mi> <mrow> <mo>(</mo> <mi>i</mi> <mo>,</mo> <mi>j</mi> <mo>)</mo> </mrow> <mo>|</mo> </mrow> <mn>2</mn> </mfrac> </mrow>

wherein m is₁(i, j) and m₂(i, j) respectively representing the gray values of two points which are closest to the gray value of a (i, j) in the neighborhood of the pixel point (i, j);

if H (i, j) ≧ T₁And V (i, j) ≧ T₂，T₁And T₂If the set threshold value is represented, processing the noise point by adopting median filtering, otherwise, processing the noise point by adopting mean filtering;

s1.2: carrying out histogram equalization on the gesture gray level image subjected to denoising in the step S1.1;

s1.3: establishing a cell neural network model of the gesture gray level image, and taking the gray level value of each pixel point (i, j) of the equalized gesture gray level image as the input u of the corresponding cell in the cell neural network model_ijIterating according to a formula of a state transition process until the whole cell neural network converges to obtain the output yi of each cell_j(t); traversing the output value of the cell corresponding to each pixel point in the cellular neural network, and when the output value of a certain pixel point is [0,1]]In the range, if the sum of the pixel values of other pixel points in the corresponding neighborhood is greater than a preset threshold value, the pixel is not an edge pixel, otherwise, the pixel is an edge pixel point; when the output value is in the range of [ -1,0), the pixel is not an edge pixel;

s1.4: obtaining a connected region according to the edge pixel points obtained in the step S1.3, extracting the outline of the connected region, and respectively performing fingertip detection on each connected region, wherein the fingertip detection method comprises the following steps:

traversing each contour pixel point in the communication area, taking the pixel point as a reference point, and marking the coordinate as p (p)_x,p_y0), a distance constant L is preset, and the L-th point p before the p point is taken along the contour direction₁(p_1x,p_1y0), take the lth point p after point p₂(p_2x,p_2y0), calculating a vectorAnd vectorIncluded angle therebetweenThe cosine value cos α, if cos α is larger than a preset curvature threshold value T, the point is judged to be a fingertip point to be determined, otherwise, the point is not judged to be a fingertip point to be determined;

determining fingertip position vector product according to traversal directionIf the gesture area overall contour is clockwise traversed, the vector product sign is negative, otherwise, the vector is positive, and the pointed point vector to be determined is calculatedAnd vectorCross product ofIf the sign of the vector product is the same as the sign corresponding to the fingertip position, reserving the vector product as a fingertip point to be determined, otherwise, not reserving the vector product;

judging whether the difference value of the y coordinate of the undetermined fingertip point with the maximum y coordinate and the y coordinate of the undetermined fingertip point with the minimum y coordinate in all the fingertip points to be determined detected in the connected region exceeds half of the height of the human face, if so, judging that the connected region is not a gesture region, otherwise, judging that the connected region is a gesture region to be determined; further judging whether the number of the pointed points to be determined in each region to be determined is larger than a preset number threshold, if so, determining that the connected region is a gesture region, otherwise, not;

solving a main direction of the gesture area, and dividing the gesture area according to the main direction and the gesture length-width ratio of 2 to obtain a divided gesture area;

s1.5: representing the contour point coordinates of the gesture area in a complex form by using the gesture area obtained by the segmentation in the step S1.4, forming a discrete sequence by using all the contour point coordinates, recording the number of contour points as n, carrying out Fourier transform on the discrete sequence to obtain n Fourier coefficients z (k), wherein k is 0,1, …, n-1, and calculating a Fourier descriptor

Wherein k ═ 1,2, …, n-1,representing the included angle between the main direction of the gesture area and the x axis;

selecting the first Q constructed feature vectors in a Fourier descriptor;

s2: inputting the feature vectors of the gesture images of the training samples into a BP neural network as training samples, and training the BP neural network by using the corresponding gesture image categories as the output of the BP neural network;

s3: and inputting the feature vectors of the gesture images to be recognized into the BP neural network trained in the step S2, and outputting the recognized gesture image categories.

2. Gesture recognition method according to claim 1, characterized in that in step S1.1 the threshold T is₁＝3.5_ij，_ijRepresenting the average absolute deviation of all the pixel points in the denoising window of the pixel point (i, j); threshold value T₂Is an integer of 6 to 10.

3. The gesture recognition method according to claim 1, wherein in step S1.3, the feedback template a in the cellular neural network is:

<mrow> <mi>A</mi> <mrow> <mo>(</mo> <mi>k</mi> <mo>,</mo> <mi>l</mi> <mo>)</mo> </mrow> <mo>=</mo> <mfenced open = "{" close = ""> <mtable> <mtr> <mtd> <mrow> <mi>a</mi> <mo>,</mo> </mrow> </mtd> <mtd> <mrow> <mi>k</mi> <mo>=</mo> <mi>i</mi> <mo>,</mo> <mi>l</mi> <mo>=</mo> <mi>j</mi> </mrow> </mtd> </mtr> <mtr> <mtd> <mrow> <mn>0</mn> <mo>,</mo> </mrow> </mtd> <mtd> <mrow> <mi>k</mi> <mo>&amp;NotEqual;</mo> <mi>i</mi> <mo>,</mo> <mi>l</mi> <mo>&amp;NotEqual;</mo> <mi>j</mi> </mrow> </mtd> </mtr> </mtable> </mfenced> <mo>,</mo> </mrow>

the control template B is:

<mrow> <mi>B</mi> <mrow> <mo>(</mo> <mi>k</mi> <mo>,</mo> <mi>l</mi> <mo>)</mo> </mrow> <mo>=</mo> <mfenced open = "{" close = ""> <mtable> <mtr> <mtd> <mrow> <mi>b</mi> <mo>,</mo> </mrow> </mtd> <mtd> <mrow> <mi>k</mi> <mo>=</mo> <mi>i</mi> <mo>,</mo> <mi>l</mi> <mo>=</mo> <mi>j</mi> </mrow> </mtd> </mtr> <mtr> <mtd> <mrow> <mo>-</mo> <mi>c</mi> <mo>,</mo> </mrow> </mtd> <mtd> <mrow> <mi>k</mi> <mo>&amp;NotEqual;</mo> <mi>i</mi> <mo>,</mo> <mi>l</mi> <mo>&amp;NotEqual;</mo> <mi>j</mi> </mrow> </mtd> </mtr> </mtable> </mfenced> <mo>,</mo> </mrow>

the threshold value I is-d,

wherein (k, l) is neighborhood N with cell C (i, j) in cellular neural network as center and side length of 2r +1_rThe points in (i, j), a, b, c, d are all normal numbers.

4. The gesture recognition method according to claim 1, wherein in step S1.4, the method for obtaining the main direction of the gesture area comprises: and (3) solving the mass center of the gesture area, then solving the vectors of the mass center to each fingertip point, and averaging the vectors, wherein the direction of the average vector is the main direction of the gesture area.