CN103376890B - The gesture remote control system of view-based access control model - Google Patents

Abstract

The invention discloses a gesture remote control system based on vision, which includes: an image capture device for capturing a series of images of an object; a gesture recognition device for a series of images captured by the image capture device The image recognizes the gesture of the object and sends the recognition result to the operation command triggering device; and the operation command triggering device triggers a predetermined operation command according to the recognition result sent from the gesture recognition device. The gesture recognition device includes: a hand detection part for detecting a hand of a subject from an image captured by an image capture device; a hand tracking part for detecting a hand of a subject in an image by the hand detection part, Track the object's hand in the image that comes down; The gesture recognition part is used to determine the motion of the object's hand according to the hand of the object detected by the hand detection part and the hand of the object tracked by the hand tracking part and determine the movement of the object's hand according to the determined object's hand motion to recognize gestures of objects.

Description

Vision-based gesture remote control system

technical field

The present invention relates to the fields of image processing, pattern recognition and object tracking, and more particularly to vision-based gesture remote control systems.

Background technique

As computers and numerous portable smart devices become more and more indispensable in people's lives today, people will desire more natural and efficient interactions between humans and computers. However, traditional human-computer interaction (HCI) peripherals such as mouse/keyboard, remote control, and even touch screens are difficult for users under some special conditions (for example, in the bathroom or kitchen, while driving, etc.) It is inconvenient for those places, because what is needed in these places is touch-free HCI. Therefore, in recent years, the gesture remote control system as one of the potential solutions has received more and more attention.

Basically, the gesture remote control system will track the hands and analyze meaningful hand expressions, if they are recognized as one of the pre-defined gestures, the corresponding operation command will be triggered to perform the predetermined operation. Since gesture recognition is complex in many situations, many different tools are employed to solve this problem in gesture recognition processing, such as Hidden Markov Models (HMM), particle filters, finite State Machines (FSM) and Neural Networks. Most gesture recognition systems require high computational complexity; moreover, some of them have certain limitations, for example, require additional equipment (e.g. need to wear gloves) or sophisticated instrumentation (e.g. need infrared camera to collect depth information) or can only be used in Operate in a well-lit environment and in a simple background environment (such as not being able to distinguish between a hand and an object with a similar color to the skin tone, or only recognizing static gestures, etc.).

Therefore, there is a need for a real-time remote control with low computational complexity and a gesture recognition system that can operate well in complex environments.

Contents of the invention

According to an aspect of the present invention, a vision-based gesture remote control system includes: an image capture device for capturing a series of images of an object; a gesture recognition device for capturing a series of images captured by the device to recognize the gesture of the object and sending the recognition result to the operation command triggering device; triggering a predetermined operation command, wherein the gesture recognition device includes: a hand detection part for detecting a subject's hand from an image captured by the image capture device; a hand tracking part for using for tracking the subject's hand in a subsequent image when the hand detection component detects the subject's hand in one image; a gesture recognition component for detecting The hand of the subject and the hand of the subject tracked by the hand tracking part determine the movement of the hand of the subject and recognize the gesture of the subject based on the determined movement of the hand of the subject.

In one embodiment, said hand detection means detects the subject's hand from a grayscale image using a local binary pattern based cascade classifier by transforming the image captured by said image capture device into a grayscale image.

In one embodiment, the hand tracking component tracks the subject's hand by using the range of the hand detected or tracked in the previous image and the distance between the skin color image of the current image and the skin color image of the previous image. difference image, to initially define the search range for tracking the hand in the skin color image of the current image; execute the template matching method to determine the range of the hand as the tracked current image, wherein the template matching method includes:

A plurality of first candidate hand ranges are defined in the search range, and the first candidate hand ranges have the same size as the target template, and second candidate hand ranges are defined in the difference image, and the candidate hand ranges have the same size as the target template. The size of the template is the same size, wherein the target template is the range of the hand detected or tracked in the previous image;

For the multiple first candidate hand ranges, perform the following steps in a loop until the multiple first candidate hand ranges have undergone the following matching judgment processing, so as to determine the candidate hand range that best matches the target template as the skin color image in the current image The range of hands tracked in:

Calculate the average value of the absolute difference of each pixel of the range of a first candidate hand and the target template as the first error;

If the first error is greater than the first predetermined threshold, it means that the range of the candidate hand does not match the target template, thereby being excluded;

If the first error is less than a first predetermined threshold, a second error is calculated, the second error being the value obtained by subtracting the average value of the values of the pixels of the second candidate hand range from the first error by a predetermined adjustment The value obtained by the coefficient;

If the second error is less than the second predetermined threshold, it is determined to match, that is, the first candidate hand range is determined to be the range of the hand tracked in the skin color range of the current image, and the value of the second error is used as the second threshold In order to make a matching judgment on the range of the next first candidate.

In one embodiment, after determining the search range in the skin color image of the current image and before performing the template matching method, refer to the difference image to modify the initially defined search range, and within the reduced search range defining the plurality of candidate hand ranges and performing the template matching method to determine the range of the hand in the current image, wherein the correction includes: gradually shrinking each side of the initially defined search range inward, and when any side meets When a pixel whose pixel value is greater than a predetermined threshold is reached, the edge stops shrinking inward.

In one embodiment, in the template matching method, when the range of each first candidate is determined to match the target template, the match is verified, and the verification includes: calculating the first candidate that is currently confirmed as the best match The average value of the absolute difference of each pixel of the hand range and the range of the hand detected in the hand detection device is used as the third error; whether the third error is judged to be greater than the third threshold, if the third error is greater than the third If the predetermined threshold is determined, it is determined that the first candidate hand range that is confirmed to be the best match does not actually match, thereby excluding the first candidate hand range.

In one embodiment, the skin color image used in the present invention can be obtained through the following steps: the value of the R component in the RGB components of each pixel of the captured image is subtracted from the mean value of the G component and the B component to obtain Obtain a difference; and compare the difference with a predetermined threshold, if the difference is smaller than the predetermined threshold, the value of the corresponding pixel of the skin color image is 0, and if the difference is smaller than the predetermined threshold is large, the value of the corresponding pixel of the skin color image takes the difference.

In one embodiment, the gesture recognition component is calculated from the hand position of each frame image according to the hand position of the object detected or tracked by the hand detection component and the hand tracking component in each frame image The hand position and orientation between two adjacent frames of images and the displacement direction of two adjacent frames of images calculated from the adjacent two hand positions and orientations are used to determine the motion of the object's hand and compare the determined motion of the object's hand with the predetermined The defined gesture-hand motion trajectory mapping table is compared to recognize the gesture of the object.

According to another aspect of the present invention, a vision-based gesture remote control method includes capturing a series of images of an object; recognizing the gesture of the object from the captured series of images; triggering a predetermined operation command according to the recognition result, wherein the recognition of the object The gesture includes: detecting the subject's hand from the captured image; when the subject's hand is detected in one image, tracking the subject's hand in the next image; and based on the detected subject's hand and the tracked The hand of the subject is used to determine the movement of the hand of the subject and the gesture of the subject is recognized based on the determined movement of the hand of the subject.

According to yet another aspect of the present invention, a method for converting an RGB image into a skin color image, the method includes: subtracting the mean value of the G component and the B component from the value of the R component of the RGB component of each pixel of the RGB image , to obtain a difference; compare the difference with a predetermined threshold, if the difference is smaller than the predetermined threshold, the value of the corresponding pixel of the skin color image is 0, and if the difference is smaller than the predetermined threshold If the threshold is large, the value of the corresponding pixel of the skin color image takes the difference.

According to yet another aspect of the present invention, a method for tracking an object in a sequence of images includes: using the range of an object detected or tracked in a previous image and the skin color image of the current image and the skin color image of the previous image Between the difference image, to initially define the search range for tracking the target in the skin color image of the current image; execute the template matching method to determine the range of the target as the tracked current image, wherein the template matching method includes:

A plurality of first candidate target ranges are defined in the search range, the first candidate target ranges have the same size as the target template, and a second candidate target range is defined in the difference image, the candidate target ranges the range has the same size as the size of the target template, wherein the target template is the range of the target detected or tracked in the previous image;

For the multiple first candidate target ranges, the following steps are cyclically executed until the multiple first candidate target ranges have undergone the following matching judgment processing, so as to determine the candidate target range that best matches the target template as the skin color image in the current image Range of targets tracked in :

Calculating the average value of the absolute difference between the first candidate target range and each pixel of the target template as the first error;

If the first error is greater than the first predetermined threshold, it means that the candidate target range does not match the target template, and thus is excluded;

If the first error is smaller than a first predetermined threshold, a second error is calculated, the second error being a value obtained by subtracting the average value of the values of the pixels of the second candidate target range from the first error by a predetermined adjustment The value obtained by the coefficient;

If the second error is smaller than the second predetermined threshold, match is determined, that is, the first candidate target range is determined to be the range of the target tracked in the skin color range of the current image, and the value of the second error is used as the second threshold In order to perform matching judgment on the next first candidate target range.

According to the vision-based gesture recognition system of the present invention, it is simple and efficient to accurately detect the hand, track Hand movement and gesture recognition.

Description of drawings

Fig. 1 shows the schematic structure of the gesture control system of the present invention;

Fig. 2 shows the flowchart of the gesture recognition processing performed by the gesture recognition device of the present invention;

Figure 3 shows examples of captured RGB images and transformed grayscale images;

Figure 4 shows an example when a hand is successfully detected in the transformed grayscale image, where the extent of the hand is shown by a rectangular box;

Figure 5 shows an exemplary diagram of a skin tone image utilized in the gesture recognition process of the present invention;

Fig. 6 shows the difference image between the skin tone images of two consecutive images for hand movement to the right;

(a), (b) and (c) in Fig. 7 show the exemplary diagram of the process of defining the search range for tracking the hand in the skin color image of the current image in the gesture recognition process of the present invention;

Fig. 8 shows an exemplary diagram of the result after the initially defined search range is revised;

Figure 9 shows an exemplary diagram of the process of tracking hands across the face by utilizing an improved template matching algorithm in the gesture recognition process of the present invention; and

FIG. 10 shows an exemplary diagram for calculating hand position orientation, hand position orientation change, and hand motion direction defined by hand position orientation in the gesture recognition device of the present invention.

detailed description

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. Note that, in this specification and the appended drawings, structural elements that have substantially the same function and structure are denoted with the same reference numerals, and repeated description of these structural elements is omitted.

The present invention provides an efficient and robust vision-based gesture remote control system that works with common web cameras and exhibits reliability in complex environments while requiring less computation.

FIG. 1 is a diagram showing a schematic structure of a gesture control system of the present invention. As shown in Figure 1, the vision-based gesture remote control system of the present invention mainly includes three parts: an image capture device 101, which is used to capture a series of images of objects; a gesture recognition device 102, which is used to capture some The column image recognizes the gesture of the object and sends the recognition result to the operation command triggering device 103 ;

In this specification, a web camera is used as the image capture device, but the present invention is not limited thereto, and any type of known or future known capture device capable of capturing video images can be used as the image capture device . In the present invention, the gesture recognition device is the core part of the gesture control system, which recognizes meaningful gestures and compares the recognized gestures with predefined gestures. match, the recognition result is sent to the operation command triggering device to trigger a predetermined operation command. The content of gesture recognition processing of the present invention will be described in detail below.

Before describing the gesture recognition processing of the present invention, the predefined gestures supported by the gesture recognition processing are given first, as shown in Table 1. It can be seen from Table 1 that compared with static gestures, the gesture recognition method of the present invention also supports dynamic gestures. For example, when the hand "pauses", the hand remains still for a very short period of time (e.g. less than 1 second), when the hand remains "pause", the hand remains still for more than 1 second, where "1 second" is just an example And this value can be set arbitrarily by the user. In addition, for example, when the hand moves to the left (right, up, down), the hand motion trajectory is to move to the left (right, up, down) after a short stay, while when the hand is waving, the hand motion trajectory for left-right-left-down (ie, →, ←, →, ←) movement. In addition, when the hand rotates clockwise, the trajectory of the hand is And when the hand rotates counterclockwise, the trajectory of the hand is Here, the hand motion trajectory may be a complete circle or a half circle. Furthermore, if the user does not make any meaningful gestures, arbitrary hand motions are recorded. Although the above gesture definition is given here, the present invention is not limited thereto, and the user can define it arbitrarily according to needs, so the gestures that can be used for recognition can be more natural.

Table 1: Gesture definitions

Next, a specific procedure of gesture recognition processing of the present invention will be described. The gesture recognition process detects the subject's hand from the captured image, tracks the movement of the hand, and then recognizes what meaningful gesture commands are expressed according to the movement trajectory of the hand and the predefined gestures. The specific process is shown in Figure 2 Show.

FIG. 2 is a flowchart showing gesture recognition processing performed by a gesture recognition device of the present invention. As shown in Figure 2, the gesture recognition processing of the present invention consists of three main processing stages: detection, tracking and recognition. Specifically, for each image frame in a series of images of the object captured by the image capture device, the following process is performed. Firstly, in S201, the image of the object captured by the image capture device is normalized, for example, it is normalized to 160 pixels×120 pixels. Of course, the present invention is not limited thereto, and the normalized size can be set arbitrarily by the user. In S202, determine whether the hand is detected and whether the hand is tracked in the previous frame image according to the records of hand detection and hand tracking, if no hand is detected in the previous frame, then the processing of hand detection is performed, if the hand is detected in the previous frame If the hand has been tracked in the frame, the processing related to hand tracking is performed. For example, in one embodiment, the flag isTrack is set to indicate whether a hand is detected, isTrack is assigned 1 when a hand is detected, and isTrack is assigned 0 when a hand is not detected. In addition, in this embodiment, the flag isDectec is set to indicate whether the hand is tracked. When the hand is tracked, isDectec is assigned 1, and when the hand is not tracked, isDectec is assigned 0. Therefore, in this embodiment, it can be determined whether a hand was detected and tracked in the previous frame by checking the values of the flags isTrack and isDectec.

In the case where a hand was not detected and tracked in the previous frame, that is, NO in S202 , processing regarding hand detection is performed. Specifically, as shown in FIG. 2, in S203, the current image is converted from an RGB image to a grayscale image. Of course, if the captured image is a grayscale image, this step will be omitted. Figure 3 shows examples of captured RGB images and transformed grayscale images. In S204, a pre-trained cascade classifier based on LBP (Local Binary Pattern) is used to detect sidehands in the entire range of the transformed grayscale image. Fig. 4 shows an example when a hand is successfully detected in the transformed grayscale image, where the extent of the hand is shown with a rectangular box. As mentioned above, if a hand is detected, isDectec is assigned 1 in S206, and if no hand is detected, isDectec is assigned 0 in S207, and these records are saved for use when processing the next frame of image.

When no hand is detected in the converted grayscale image, the gesture recognition method of the present invention returns to S201 to process the next frame of image. On the other hand, when the hand is successfully detected in the converted grayscale image, the current RGB image is transformed into a skin color image in S208, which is convenient for separating the skin color area from the background, thereby reducing the need for processing the next frame image. Effects due to background during hand tracking processing. More specifically, the method for converting an RGB image into a skin color image used in the present invention is as follows: assuming that the value of a pixel in the skin color image is represented by s, the R component, the G component, and the RGB component of each pixel in the RGB image are The values of the B components are respectively represented by r, g, and b, and a temporary intermediate variable Temp is set, then the value of s is defined by the following formula:

Temp=r-((g+b)/2));

Temp = MAX(0,Temp);

s=Temp>140? 0: Temp;

That is, at first, subtract the mean value of the G component and the B component from the value of the R component in the RGB components of each pixel of the RGB image to obtain a difference; then, compare the difference with a predetermined threshold, if If the difference is smaller than a predetermined threshold, the value of the corresponding pixel of the skin-color image takes 0, and if the difference is larger than the predetermined threshold, the value of the corresponding pixel of the skin-color image takes the difference. For example, FIG. 5 shows an example of a skin color image after the captured RGB image is transformed with a predetermined threshold 140, where a rectangular box indicates that a hand is detected, and the skin color area is separated from the background as described above. Although the method for obtaining a skin color image from an RGB image as described above is described here, the user may also use other skin color separation methods to obtain a skin color image as required. In addition, the skin color image of the current image is saved for differential image calculation when performing hand tracking processing on the next image, and parameters representing the range of detected hands are also saved for use in hand tracking processing for the next image. Template matching and error evaluation during tracking processing.

In the case where a hand has been detected or tracked in the previous image, that is, YES in S202, processing regarding hand tracking is performed. The hand tracking process of the present invention uses an improved template matching method that is computationally faster and can tolerate some changes in hand shape during hand motion. Specifically, in S209, the current image is converted from an RGB image to a skin color image, and the conversion method is as described above. In S210, calculate the difference image between the skin color image of the current image and the saved skin color image of the previous image. In the calculation of the difference image, only the difference values along the hand motion direction are kept, while the values opposite to the hand motion direction are discarded. Specifically, firstly, the difference between the value of each pixel of the skin color image of the current image and the corresponding pixel value of the skin color image of the previous image is calculated. Next, compare the difference with 0, if the difference is greater than 0, the value of the corresponding pixel in the difference image takes the difference, and if the difference is smaller than 0, the value of the corresponding pixel in the difference image takes 0. Fig. 6 shows the difference image between the skin tone images of two consecutive images for hand movement to the right.

Next, in S210, the hand is tracked in the skin color image of the current image according to the hand detected or tracked in the previous image and the difference image between the skin color image of the current image and the skin color image of the previous image. Specifically, firstly, the range of the hand detected or tracked in the previous image is used to initially define the search range for tracking the hand in the skin color image of the current image. For example, in the example where the range of the hand is shown by a rectangular frame, the search range can be defined according to the following formula:

Range.x=MAX(0, Target.x-15);

Range.y=MAX(0, Target.y-15);

Range.width=MIN(a-Range.x, Target.width+30);

Range.height=MIN(b-Range.y, Target.height+30),

Among them, Target.x and Target.y represent the horizontal and vertical coordinates of the vertices of the upper left corner of the rectangular frame of the hand detected or tracked in the previous image, and Target.width and Target.height represent the detected or tracked hand in the previous image The width and height of the rectangular frame of the hand, and Range.x and Range.y represent the abscissa and ordinate of the vertex in the upper left corner of the search range in the skin color image of the current image, and Range.width and Range.height represent the height of the current image The width and height of the search range in the skin color image, and a and b represent the horizontal and vertical pixel numbers of the image respectively, and the numerical values of "15", "30" are set according to empirical values and can be such that Initially set the search range as close as possible to other values of the tracked hand range.

In order to reduce the amount of calculation in the template matching method in the next step and improve the accuracy, after initially defining the search range as described above, the difference image between the skin color image of the current image and the skin color image of the previous image is used to correct the initial The defined search range refers to the region of interest in the difference image as the corrected search range. As shown in Figure 7, it shows (a) the extent of the hand detected or tracked in the previous image, (b) the difference image between the skin color image of the current image and the skin color image of the previous image, and (c) The search range defined in the skin color image of the current image, wherein the label at the upper left corner in the figure exemplarily represents the frame number of the image. And, FIG. 8 shows a diagram of search range reduction as an example of search range modification, wherein the four sides of the initially defined search range are gradually shrunk inward, and when any side encounters a pixel value greater than a predetermined threshold (eg 5) pixels, then this side stops shrinking inwards.

After the search range is defined in the skin color image of the current image, in S212, a template matching method is performed to determine the range of the hand as the tracked current image. The template matching method here refers to using the range of the hand detected or tracked in the previous image as the target template, comparing the range of the candidate hand with the template, and determining the match if the error is less than a certain value. To reduce the tracking error when the hand crosses some objects with skin color (e.g., face, another hand), the above template matching method considers the motion information from the difference image, specifically:

A plurality of first candidate hand ranges are defined in the search range, and the first candidate hand ranges have the same size as the target template, and second candidate hand ranges are defined in the difference image, and the candidate hand ranges have the same size as the target template. The size of the template is the same size as;

For the multiple first candidate hand ranges, perform the following steps in a loop until the multiple first candidate hand ranges have undergone the following matching judgment processing, so as to determine the candidate hand range that best matches the target template as the skin color image in the current image The range of the hand tracked in: Calculate the average value of the absolute difference between the first candidate hand range and each pixel of the target template as the first error; if the first error is greater than the first predetermined threshold, it means that the candidate hand range Does not match the target template, thereby being excluded; if the first error is less than the first predetermined threshold, then calculate the second error, the second error is by subtracting the first error from the average value of each pixel in the second candidate hand range If the second error is smaller than the second predetermined threshold, it is determined to match, that is, the first candidate hand range is determined to be the hand tracked in the skin color range of the current image. range, and the value of the second error is used as a second threshold to make a matching judgment on the range of the next first candidate.

As described above, since the present template matching method finds the best matching hand range step by step by referring to the previous candidate hand range, this makes it possible to tolerate changes in the hand shape of the hand in continuous motion. However, this also leads to the accumulation of matching errors during successive tracking processes. Therefore, the historical matching error can be limited by setting an additional verification process, which includes: calculating the difference between the range of the first candidate hand currently confirmed as the best match and the range of the hand detected in the hand detection device The average value of the absolute difference is used as the third error; it is judged whether the third error is greater than the third threshold, and if the third error is greater than the third predetermined threshold, then it is determined that the currently most matched first candidate range does not actually match, Thus, the first candidate range is excluded. Fig. 9 shows the tracking results when the hand moves across the face, in order to show the effect more clearly, alternate image frames are selectively shown, and the rectangular boxes represent the tracked hands.

Next, in S213, it is judged whether the hand is successfully tracked in S212. If the hand is successfully tracked, the flag isTrack is set to 1 as described above in S214, otherwise the flag isTrack is set to 0 in S215 and the method flow goes to S203 to perform hand detection processing on the current image.

In the case where the flag isTrack is assigned 1 in S214, that is, in the case where the hand is tracked in the current image, in S216, the skin color of the current image is saved for the differential image calculation when the next frame image is processed, and Save the parameters representing the tracked hand range in the current image for template matching processing on the next frame image, and define the horizontal and vertical coordinates of the tracked hand position in the current image as Or the sum of the abscissa and half of the width of the range of the tracked hand and the sum of the ordinate and half of the height of the range of the hand detected or tracked in the previous image, if HandPos(x, y) is used to represent the current The coordinates of the tracked hand position in the image are expressed as the following formula:

HandPos(x, y) = (Target.x+Target.width/2, Target.y+Target.height/2)

Next, at S217 , after hand detection or hand tracking is performed on all the images in the series of images captured by the image capture device, the hand movement trajectory can be obtained through the recorded multiple hand positions. The gesture of the object can be recognized by analyzing the motion trace and referring to the defined gesture-motion trace mapping table. Specifically, first, calculate the hand position orientation Orient between the hand positions detected or tracked in every two adjacent images, that is, calculate the difference between the hand position detected or tracked in the current image and the hand position detected in the previous image. Or the angle between the line formed by the tracked hand position and the horizontal direction

Next, calculate the displacement direction DeltaOrient between two adjacent images. Specifically, the hand position orientation Orient of the current image relative to the previous image is subtracted from the hand position orientation LastOrient of the subsequent image relative to the current image to obtain a difference. If the absolute value of the difference is greater than 180, then in the case of the difference greater than 0, subtract 360 from the difference to obtain a value as the image displacement direction DeltaOrient; in the case of the difference less than 0, the Add 360 to the difference value to obtain a value as the image displacement direction DeltaOrient. If the absolute value of the difference is not greater than 180, the difference is used as the image displacement direction DeltaOrient.

Then, for 8 meaningful gestures ("any" is used to record real-time motion traces, and "short stay" is set as a preparatory action for up/down/left/right gestures, stay, up/ Down/left/right, waving as a group, they are based on the value of Orient; CW/CC as another group, they are based on the value of DeltaOrient and the cumulative sum of a period of time, in particular, for the waving gesture, which includes Alternating motion of left and right motions), the gesture recognition is as follows:

stay

In order to recognize the gesture of staying, it needs to detect that the hand remains still for consecutive STAY_NUM frames, that is, for consecutive STAY_NUM frames, there is always the same hand position tracked in the current image as the detected or tracked hand position in the previous image, Where STAY_NUM is a predefined number of frames.

up/down/left/right

In order to recognize up/down/left/right gestures, first, the preparatory action "short stay" should be detected, then in the next DIREC_NUM frame, the value of Orient should be kept within a certain range along one direction, where, DIREC_NUM is a predefined number of frames. As an example, assume a "stop" action of staying still for 3 frames. Fig. 10(b) shows the range of values of Orient for up/down/left/right directions. For example, if 46≤Orient≤134, it is considered to be moving in an upward direction.

wave:

In order to recognize the waving gesture, it is necessary to detect consecutive 4 motion segments: right-left-right-left. In order to detect each motion segment, it is required that the Orient value should be kept within a corresponding value range (MIN_SEG_NUM≤N<=MAX_SEG_NUM) for consecutive N frames, where MIN_SEG_NUM and MAX_SEG_NUM are predefined frame number thresholds.

clockwise rotation:

In order to recognize the gesture of clockwise rotation, the value of DeltaOrient needs to be kept positive in several consecutive frame images, the absolute value of DeltaOrient is kept in a certain range (for example, a range greater than 10 and less than or equal to 50) and the sum of the absolute values of DeltaOrient The predetermined threshold CIRCLE_DEGREE has been reached.

Anticlockwise rotation:

In the judgment of counterclockwise rotation, except that the value of DeltaOrient remains negative, the others are the same as the judgment of clockwise rotation.

After a gesture is recognized as described above, all counters for the corresponding gesture candidate will be reset. Also, the system will be "frozen" for a few frames, i.e. the gesture will not be recognized during these few frames. This mechanism is introduced to avoid false recognition of some unintentional gestures. For example, if a user wishes to continuously gesture to the right, he or she will naturally move the hand back after completing one gesture to the right before making the next gesture to the right. If this "freeze" mechanism is not utilized, this "move back" will likely be misrecognized as a gesture to the left.

In this way, the gesture of the subject is recognized.

Next, in S219, the gesture as the recognition result is output to the operation command triggering device.

Then, in the operation command triggering device as an interface between the gesture and the corresponding operation command, the corresponding operation command is triggered by looking at a predefined mapping table of gestures and operation commands according to the gesture recognition result sent from the gesture recognition device. A mapping table of gestures and operation commands is shown in Table 2 below, for example. The table 2 shows the mapping between the operations of the windows image viewer software corresponding to the 8 gestures. Although the mapping between the operations of the windows picture viewer software corresponding to the 8 gestures is given here, the present invention is not limited thereto, and the mapping between more gestures and corresponding operations can be defined, and the operations It may involve not only the operation of various types of software, but also the operation of different functions of various electronic devices, which is defined by the user according to needs. For example, the invention can be applied to gesture-mouse control.

gesture operation command short stay carriage return left Previous/Move to the left if the picture is zoomed in To the right Next / If the picture is enlarged, it means move to the right up If the picture is zoomed in, it means move up down If the picture is zoomed in, it means move down to wave quit clockwise rotation enlarge Anticlockwise rotation zoom out

Table 2: Gesture-operation command mapping table

According to the vision-based gesture recognition system of the present invention, it is simple and efficient to accurately detect the hand, track Hand movement and gesture recognition.

The above describes the components of the vision-based gesture recognition remote control system according to the present invention and the corresponding operations in each component, but various changes and modifications can be made without departing from the gist of the present invention, and these changes and modifications also fall within within the scope of this application.

Claims (9)

1. A gesture remote control system based on vision, comprising: an image capture device for capturing a series of images of the subject; a gesture recognition device for recognizing a gesture of an object from a series of images captured by the image capture device and sending the recognition result to the operation command trigger device; and An operation command triggering device, the operation command triggering device is used to trigger a predetermined operation command according to the recognition result sent from the gesture recognition device, wherein the gesture recognition device includes: hand detection means for detecting a subject's hand from an image captured by the image capture device; hand tracking means for tracking the subject's hand in a subsequent image when the hand detection means detects the subject's hand in an image; a gesture recognition part configured to determine the motion of the subject's hand based on the subject's hand detected by the hand detection part and the subject's hand tracked by the hand tracking part and motion to recognize gestures of objects, Wherein, the hand tracking component tracks the object's hand through the following processing: Use the range of the hand detected or tracked in the previous image and the difference image between the skin color image of the current image and the skin color image of the previous image to initially define the search range for tracking the hand in the skin color image of the current image ; perform a template matching method to determine the extent of the hand as the tracked current image, Wherein, the template matching method includes: A plurality of first candidate hand ranges are defined in the search range, and the first candidate hand ranges have the same size as the target template, and second candidate hand ranges are defined in the difference image, and the candidate hand ranges have the same size as the target template. The size of the template is the same size, wherein the target template is the range of the hand detected or tracked in the previous image; For the multiple first candidate hand ranges, perform the following steps in a loop until the multiple first candidate hand ranges have undergone the following matching judgment processing, so as to determine the candidate hand range that best matches the target template as the skin color image in the current image The range of hands tracked in: Calculate the average value of the absolute difference of each pixel of the range of a first candidate hand and the target template as the first error; If the first error is greater than the first predetermined threshold, it means that the range of the candidate hand does not match the target template, thereby being excluded; If the first error is less than a first predetermined threshold, a second error is calculated, the second error being the value obtained by subtracting the average value of the values of the pixels of the second candidate hand range from the first error by a predetermined adjustment The value obtained by the coefficient; If the second error is less than the second predetermined threshold, it is determined to match, that is, the first candidate hand range is determined to be the range of the hand tracked in the skin color range of the current image, and the value of the second error is used as the second threshold In order to make a matching judgment on the range of the next first candidate. 2. The vision-based gesture remote control system according to claim 1, wherein the hand detection component utilizes local binary pattern based cascading by transforming the image captured by the image capture device into a grayscale image A classifier detects the subject's hand from this grayscale image. 3. The vision-based gesture remote control system according to claim 1, wherein after determining the search range in the skin color image of the current image and before performing the template matching method, the initially defined search range, and define the plurality of candidate hand ranges in the reduced search range and execute the template matching method to determine the range of the hand in the current image, wherein the correction includes: the initially defined search range Each edge shrinks inward gradually, and when any edge encounters a pixel whose pixel value is greater than a predetermined threshold, the edge stops shrinking inward. 4. The vision-based gesture remote control system according to claim 1, wherein, in the template matching method, when determining a match with a target template for each first candidate hand range, the matching is verified, and the verification include: calculating the average value of the absolute difference of each pixel between the range of the first candidate hand that is currently confirmed as the best match and the range of the hand detected in the hand detection device as a third error; It is judged whether the third error is greater than a third threshold, and if the third error is greater than a third predetermined threshold, then it is determined that the currently confirmed most matching first candidate hand range does not actually match, thereby excluding the first candidate hand range. 5. The vision-based gesture remote control system according to any one of claims 1 to 4, wherein the skin color image is obtained through the following steps: subtracting the mean value of the G component and the B component from the value of the R component in the RGB components of each pixel of the captured image to obtain a difference; and Comparing the difference with a predetermined threshold, if the difference is smaller than the predetermined threshold, the value of the corresponding pixel of the skin color image is 0, and if the difference is larger than the predetermined threshold, the color of the skin color image is The value of the corresponding pixel takes the difference. 6. The vision-based gesture remote control system according to claim 1, wherein the gesture recognition component is based on the position of the hand of the object detected or tracked by the hand detection component and the hand tracking component in each frame of image , The hand position orientation between two adjacent frame images calculated from the hand position of each frame image and the displacement direction of two adjacent frame images calculated from the adjacent two hand position orientations to determine the movement of the subject's hand And comparing the determined hand movement of the object with a predefined gesture-hand movement trajectory mapping table to identify the gesture of the object. 7. A vision-based gesture remote control method, comprising: capture a series of images of the subject; Recognize gestures of objects from a series of captured images; Trigger a predetermined operation command according to the recognition result, Among them, gestures for recognizing objects include: Detecting the subject's hand from the captured image; when the subject's hand is detected in one image, tracking the subject's hand in a subsequent image; and determining a movement of the subject's hand based on the detected subject's hand and the tracked subject's hand and recognizing a gesture of the subject based on the determined subject's hand movement, The gesture remote control method also includes: Use the range of the hand detected or tracked in the previous image and the difference image between the skin color image of the current image and the skin color image of the previous image to initially define the search range for tracking the hand in the skin color image of the current image ; perform a template matching method to determine the extent of the hand as the tracked current image, Wherein, the template matching method includes: A plurality of first candidate hand ranges are defined in the search range, and the first candidate hand ranges have the same size as the target template, and second candidate hand ranges are defined in the difference image, and the candidate hand ranges have the same size as the target template. The size of the template is the same size, wherein the target template is the range of the hand detected or tracked in the previous image; For the multiple first candidate hand ranges, perform the following steps in a loop until the multiple first candidate hand ranges have undergone the following matching judgment processing, so as to determine the candidate hand range that best matches the target template as the skin color image in the current image The range of hands tracked in: Calculate the average value of the absolute difference of each pixel of the range of a first candidate hand and the target template as the first error; If the first error is greater than the first predetermined threshold, it means that the range of the candidate hand does not match the target template, thereby being excluded; If the first error is less than a first predetermined threshold, a second error is calculated, the second error being the value obtained by subtracting the average value of the values of the pixels of the second candidate hand range from the first error by a predetermined adjustment The value obtained by the coefficient; If the second error is less than the second predetermined threshold, it is determined to match, that is, the first candidate hand range is determined to be the range of the hand tracked in the skin color range of the current image, and the value of the second error is used as the second threshold In order to make a matching judgment on the range of the next first candidate. 8. The vision-based gesture remote control method according to claim 7, wherein the skin color image is obtained through the following steps: Subtracting the mean value of the G component and the B component from the value of the R component of the RGB component of each pixel of the captured image to obtain a difference; Comparing the difference with a predetermined threshold, if the difference is smaller than the predetermined threshold, the value of the corresponding pixel of the skin color image is 0, and if the difference is larger than the predetermined threshold, the color of the skin color image is The value of the corresponding pixel takes the difference. 9. A method for tracking an object in a sequence of images, comprising: Use the range of the target detected or tracked in the previous image and the difference image between the skin color image of the current image and the skin color image of the previous image to initially define the search range for tracking the target in the skin color image of the current image ; Executing a template matching method to determine the range of the target as the tracked current image, wherein the template matching method includes: A plurality of first candidate target ranges are defined in the search range, the first candidate target ranges have the same size as the target template, and a second candidate target range is defined in the difference image, the candidate target ranges the range has the same size as the size of the target template, wherein the target template is the range of the target detected or tracked in the previous image; For the multiple first candidate target ranges, the following steps are cyclically executed until the multiple first candidate target ranges have undergone the following matching judgment processing, so as to determine the candidate target range that best matches the target template as the skin color image in the current image Range of targets tracked in : Calculating the average value of the absolute difference between the first candidate target range and each pixel of the target template as the first error; If the first error is greater than the first predetermined threshold, it means that the candidate target range does not match the target template, and thus is excluded; If the first error is smaller than a first predetermined threshold, a second error is calculated, the second error being a value obtained by subtracting the average value of the values of the pixels of the second candidate target range from the first error by a predetermined adjustment The value obtained by the coefficient; If the second error is smaller than the second predetermined threshold, match is determined, that is, the first candidate target range is determined to be the range of the target tracked in the skin color range of the current image, and the value of the second error is used as the second threshold In order to perform matching judgment on the next first candidate target range.