WO2012014590A1 - Video processing device and method - Google Patents

Abstract

The video processing device of the present invention is provided with face detection means (120) for detecting the face (210) of a person (200) from an image, range estimation means (130) for estimating a hand existence range (P1) in which a hand (220) of the person may exist on the basis of information of the detected face of the person, and hand detection means (140) for performing processing for detecting the hand of the person in the hand existence range. Accordingly, it is possible to detect a hand of a person within a video efficiently and with high precision.

Description

Video processing apparatus and method

The present invention relates to a technical field of a video processing apparatus and method for detecting a predetermined object existing in a video by performing various processes on the video, for example.

As this type of video processing device, for example, there is a device that detects a predetermined object such as a person from a captured video. When detecting an object, local image features common to the object to be detected are used. For example, Non-Patent Document 1 proposes a technique of learning SVM (Support Vector Vector Machine) by calculating a histogram in a luminance gradient direction as a feature quantity and detecting a person from a video.

Navneet Dalal and Bill Triggs, “Histograms of Oriented Gradients for Human Detection”, IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2005

However, in the technique as described above, when an object is detected, it is required to first scan the entire image. For this reason, an increase in the load due to the time until the object is detected and the detection process is unavoidable. Such an inconvenience can be suppressed to a relatively small degree when detecting an object that is easy to detect (that is, a feature is clear) such as a human face. On the other hand, when detecting objects such as human hands that are difficult to detect (for example, there are few features that can be used for detection or the shape is not constant and changes greatly depending on the situation) It becomes a big problem. That is, the above-described technique has a technical problem that it is difficult to efficiently detect various objects.

The present invention has been made in view of, for example, the above-described problems, and an object of the present invention is to provide a video processing apparatus and method capable of efficiently and accurately detecting a human hand in a video.

In order to solve the above-described problem, the video processing apparatus of the present invention detects a person's face from an image, and presence of the person's hand that can exist based on the detected face information of the person Range estimation means for estimating the range, and hand detection means for performing processing for detecting the hand of the person in the hand presence range.

In order to solve the above-described problem, the video processing method of the present invention detects a person's face from an image, and the presence of the person's hand that can exist based on the detected face information of the person A range estimation step of estimating a range; and a hand detection step of performing a process of detecting the hand of the person in the hand presence range.

The operation and gain of the present invention will be clarified from embodiments for carrying out the invention described below.

It is a block diagram which shows the structure of the video processing apparatus which concerns on 1st Example. It is a flowchart which shows operation | movement of the video processing apparatus which concerns on 1st Example. It is a top view which shows an example of an input image. It is a top view which shows notionally the face detection method. It is a flowchart which shows the estimation method of a hand presence range. It is a top view which shows the estimation method of a hand presence range notionally. It is a conceptual diagram which shows the scale estimation method of the hand which is going to detect. It is a conceptual diagram which shows the rotation angle estimation method of the hand which it is going to detect. It is a block diagram which shows the structure of the video processing apparatus which concerns on 2nd Example. It is a flowchart which shows operation | movement of the video processing apparatus which concerns on 2nd Example. It is a block diagram which shows the structure of the video processing apparatus which concerns on 3rd Example. It is a conceptual diagram which shows the some image used for learning. It is a top view which shows notionally the hand presence range estimated based on learning.

The video processing apparatus according to the present embodiment includes a face detection unit that detects a person's face from an image, and a range for estimating a hand presence range in which the person's hand can exist based on the detected face information of the person. An estimation unit; and a hand detection unit that performs a process of detecting the hand of the person in the hand presence range.

According to the video processing apparatus according to the present embodiment, during the operation, a human face is first detected from the images constituting the video. Specifically, for example, the entire image is scanned, and a portion having a predetermined value such that the condition such as luminance can be recognized as a human face is detected as a face.

Subsequently, in this embodiment, a hand presence range in which a human hand can exist is estimated based on the detected face information. Here, “face information” refers to various information that can be read from the detected face, and includes, for example, the position, size, orientation, and the like of the face.

The human hand cannot exist at a position very far from the face due to the structure of the human body. That is, if the position of one person's face is known, the hand of one person can exist only within the range where the arm extends from there. Therefore, if the face can be detected, the range in which the hand can exist can be estimated with high accuracy based on the face information.

When the hand presence range is estimated, processing for detecting a human hand in the hand presence range is performed. That is, processing for detecting a person's hand is performed only in the hand presence range in the image. Examples of the process for detecting a hand include matching with a template image prepared in advance, a method using a luminance gradient direction histogram, a method using a boundary line shape of a skin color region, and the like. However, such a detection method is merely an example, and the hand detection method according to the present embodiment is not limited at all.

Here, if the above-described hand presence range is not estimated, it is required to perform a hand detection process on the entire image. In this case, the time required to detect a hand and the load due to the detection process increase due to the wide processing range.

However, in the video processing apparatus according to the present embodiment, as described above, the process of detecting the hand is performed only in the hand presence range in the image. Therefore, it is possible to detect a hand very efficiently. In the present embodiment, compared to the case where a hand is directly detected without detecting a face, the time until detection and the load due to the detection process are increased for the process of detecting the face. It is done. However, detection of human faces is technically mature and has already been put into practical use and can be used without increasing a new burden. For this reason, the influence of the face detection process is almost negligible.

Further, in the present embodiment, since the process of detecting the hand only in the hand presence range is performed, for example, when the hand of another person exists outside the hand presence range, the hand of another person is detected. It can prevent detecting as a hand of the person who is the object. That is, it is possible to prevent erroneous detection and accurately detect a human hand.

As described above, according to the video processing apparatus according to the present embodiment, it is possible to efficiently and accurately detect the hand of a person in the video.

In one aspect of the video processing apparatus according to the present embodiment, the range estimation means estimates a circle centered on the person's face as the hand presence range.

According to this aspect, since the hand presence range is estimated as a circle centered on the person's face, the process for estimating the hand presence range can be made extremely simple. The “circle around the face of a person” here is not limited to a circle centered on a portion located completely in the center of the face, but various circles including a face near the center. Or it is a wide concept including an ellipse.

The range in which a person's hand can exist is the range in which the person's arm extends. Then, it is considered that the portion serving as a fulcrum of the arm (specifically, near the shoulder) is located very close to the face. Therefore, for example, if a circle whose radius is the length of the person's arm is estimated as the hand presence range, the hand can be detected more efficiently. The length of the arm can be estimated from the size of the detected face, for example.

In another aspect of the video processing apparatus according to the present embodiment, the video processing apparatus further includes a learning unit that learns a relative positional relationship between the face of the person and the hand of the person, and the range estimation unit includes the learned relative The hand presence range is estimated based on the correct positional relationship.

According to this aspect, before estimating the hand presence range, the relative positional relationship of the person's face and the person's hand is learned by referring to many images in advance. Here, the “relative positional relationship” includes not only a simple positional relationship between a person's face and a person's hand, but also information regarding the size and orientation of the face and hand.

If the hand presence range is estimated based on the learning result described above, the range in which the process of detecting the hand is performed becomes a more appropriate range. Therefore, it is possible to detect the hand very efficiently. In addition, when performing a process of detecting a hand, processing is performed in order from a range in which the probability that a hand exists is high (that is, a range in which a human hand is detected with a high frequency in learning of a relative positional relationship). If this is executed, the hand can be detected more efficiently.

In another aspect of the video processing apparatus according to the present embodiment, the hand detection unit includes a scale estimation unit that estimates the size of the person's hand based on information about the person's face.

According to this aspect, when a face is detected, the size of the hand is estimated based on the detected face information. For example, if the face is detected large, it is estimated that the detected hand will be large. On the other hand, if the face is detected small, it is estimated that the detected hand will be small.

In this aspect, since the size of the hand that will be detected is estimated in advance, the process of detecting the hand can be performed extremely efficiently and accurately.

In another aspect of the video processing apparatus according to the present embodiment, the hand detection unit includes a rotation angle estimation unit that estimates a rotation angle of the person's hand based on information on the person's face.

According to this aspect, when a face is detected, the rotation angle of the hand (that is, the hand inclination) is estimated based on the detected face information. For example, the hand rotation angle is estimated to be within a predetermined range with respect to the detected face rotation angle. That is, the hand rotation angle can be estimated by utilizing the fact that the face rotation angle and the hand rotation angle are relatively close to each other.

In this aspect, since the rotation angle of the hand that will be detected is estimated in advance, the process of detecting the hand can be performed very efficiently and accurately.

In another aspect of the video processing apparatus according to the present embodiment, the scribe detection means includes a skin color area detection means for detecting a skin color area from the image, and in the skin color area in the hand presence range, The person's hand is detected.

According to this aspect, when the hand presence range is estimated, the skin color region is detected from the image before the processing for detecting the hand is performed. The skin color region is detected based on, for example, a preset skin color component. Or you may detect according to the color of the detected face.

When the skin color area is detected, a process for detecting the hand is performed on the skin color area in the hand presence range. That is, even in the hand presence range, the process for detecting the hand is not performed for the non-skin color area, and the process for detecting the hand is performed only in the skin color area in the hand presence range.

The color of a person's hand is considered to be a skin color unless gloves are worn, for example. For this reason, if the area | region of a skin color is detected, the range which performs the process which detects a hand can be narrowed more. Therefore, in this aspect, it is possible to detect a hand very efficiently.

In another aspect of the video processing apparatus according to the present embodiment, the hand detection unit compares the past image and the current image, and detects a difference region in which a difference in luminance is a predetermined value or more. A detection unit is included, and the person's hand is detected in the difference area in the hand presence range.

According to this aspect, when the hand presence range is estimated, the difference area is detected before the process of detecting the hand is performed. The difference area is an area where a difference in luminance is equal to or greater than a predetermined value by comparing a past image and a current image. Here, the “predetermined value” is a value set according to the difference in luminance between the portion where the hand is present and the portion where the hand is not present.

When the difference area is detected, a process for detecting a hand is performed for the difference area in the hand presence range. That is, the process of detecting the hand is performed only in the difference area within the hand presence range. That is, even if it is the hand presence range, the processing for detecting the hand is not performed for the region that is not the difference region, and the processing for detecting the hand is performed only in the difference region within the hand presence range.

A person's hand is considered to move much more than other objects in the image. For this reason, there is a high possibility that the position of the hand is different between the past image and the current image. Therefore, it is estimated that there is a high possibility that a hand is present in an area where the luminance difference is equal to or greater than a predetermined value (that is, an area where the luminance has greatly changed). For this reason, if the difference area is detected, the range for performing the hand detection process can be further narrowed. Therefore, in this aspect, it is possible to detect a hand very efficiently.

The video processing method according to the present embodiment includes a face detection step for detecting a person's face from an image, and a range for estimating a hand presence range in which the person's hand may exist based on the detected face information of the person. An estimation step, and a hand detection step for performing processing for detecting the hand of the person in the hand presence range.

According to the video processing method according to the present embodiment, as in the case of the video processing device of the present invention described above, first, the face of a person is detected, and in the hand presence range estimated based on the detected face information. Processing to detect the hand is performed. Therefore, it is possible to efficiently and accurately detect the hand of a person in the video.

Note that the video processing method of the present invention can also adopt various aspects similar to the various aspects of the video processing apparatus of the present invention described above.

The operation and other gains of various embodiments according to the video processing apparatus and method of the present invention will be described in more detail in the following examples.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

<First embodiment>
First, the configuration of the video processing apparatus according to the first embodiment will be described with reference to FIG. FIG. 1 is a block diagram showing the configuration of the video processing apparatus according to the first embodiment.

1, the video processing apparatus according to the first embodiment includes a video input unit 110, a face detection unit 120, a hand presence range estimation unit 130, a hand detection unit 140, and an output unit 150. ing.

The video input unit 110 inputs a video to be detected from the outside of the apparatus to the video processing apparatus. The video is input as, for example, a plurality of continuous images.

The face detection unit 120 is an example of the “face detection unit” of the present invention, and scans an input image to detect the face of a person in the image.

The hand presence range estimation unit 130 is an example of the “range estimation unit” of the present invention, and based on the detected face information (for example, the position, size, orientation, etc. of the face) The range in which a person's hand can exist is estimated.

The hand detection unit 140 is an example of the “hand detection unit” of the present invention, and performs a process of detecting a hand in the estimated hand presence range to detect a person's hand.

The output unit 150 outputs information on the detected human hand. The hand information output here is, for example, an information terminal in a public space such as another device (for example, a television, an air conditioner, a lighting device, a personal computer, a game machine, a portable device, a ticket machine, or a digital signage) according to the behavior of the hand. , Housing equipment, automotive in-vehicle equipment, etc.).

Next, the operation of the video processing apparatus according to the first embodiment will be described with reference to FIGS. 2 to 8 in addition to FIG. FIG. 2 is a flowchart showing the operation of the video processing apparatus according to the first embodiment. FIG. 3 is a plan view showing an example of the input image, and FIG. 4 is a plan view conceptually showing the face detection method.

In FIG. 2, during the operation of the video processing apparatus according to the first embodiment, the video input unit 110 first acquires an image constituting the video (step S01). In the following description, it is assumed that an image showing the face 210 and the hand 220 of the person 200 is acquired as shown in FIG.

When the image is acquired, the face detection unit 120 detects the face 210 of the person 200 in the image (step S02). For example, the face detection unit 120 scans the entire acquired image. Then, a part having a predetermined value such that the condition such as luminance can be recognized as a human face is detected as a face.

4, the face 210 is detected so as to draw a rectangle surrounding the face 210. At this time, the coordinates (x, y) of the upper left point of the rectangle surrounding the face 210 and the values of the width w and the height h of the face 210 are acquired. In addition to such parameters, various information such as the orientation of the face may be detected.

2, when the face 210 is detected, the hand presence range is estimated based on the information of the detected face 210. Hereinafter, the hand presence range estimation method will be described in detail with reference to FIGS. FIG. 5 is a flowchart showing the hand presence range estimation method, and FIG. 6 is a plan view conceptually showing the hand presence range estimation method. FIG. 7 is a conceptual diagram showing a method for estimating the scale of the hand to be detected, and FIG. 8 is a conceptual diagram showing a method for estimating the rotation angle of the hand to be detected.

In FIG. 5, the hand presence range estimation unit 130 estimates a circle centered on the detected face as the hand presence range (step S11). Specifically, a circle P1 as shown in FIG. 6 is estimated as the hand presence range.

The human hand cannot exist at a position very far from the face due to the structure of the human body. That is, if the position of the person's face is known, the hand can only exist within the range where the arm extends from there. Therefore, for example, if a circle whose radius is the length of the person's arm is estimated as the hand presence range, the hand can be detected more efficiently. Here, the radius of the circle P1 is determined according to the size of the detected face 210. More specifically, twice the height h of the face 210 is the radius of the circle P1.

When the hand presence range is estimated, the scale of the hand that will be detected is estimated based on the size of the detected face 210 (step S12). The hand scale is estimated to be in the range of 0.7 to 1.5 times the height h of the detected face 210, for example. When the hand scale is estimated, for example, as shown in FIG. 7, a five-step template (ie, A to O) is set.

Subsequently, based on the detected rotation angle of the face 210, the rotation angle of the hand that will be detected is estimated (step S13). The rotation angle of the hand is estimated to be in the range of +45 degrees to −45 degrees with respect to the detected rotation angle of the face 210, for example. When the rotation angle of the hand is estimated, for example, as shown in FIG. 8, a seven-step template (ie, “i”) is set.

When the hand scale and the rotation angle are estimated, it is determined whether or not the processing for all the faces in the image (that is, the processing from step S11 to step S13) has been completed (step S14). If it is determined that the processes for all the faces have not been completed (step S14: NO), the processes from step S11 to step S13 are performed again for the faces for which the process has not been completed. On the other hand, if it is determined that the processing for all the faces has been completed (step S14: YES), a combination of a range for actually performing hand detection, a scale, and a rotation angle is set (step S16). That is, the parameters estimated in the processing from step S11 to step S13 are combined with each other, and a condition for actually detecting the hand is set.

Returning to FIG. 2, when the condition for detecting the hand is set as described above, the hand detecting process is performed in the hand detecting unit 140 (step S04). The hand detection unit 140 detects the hand 220 in the image by matching with a template image prepared in advance (see FIGS. 7 and 8). Note that it is also possible to perform hand detection using a method using a luminance gradient direction histogram or a method using a boundary line shape of a skin color region.

Here, if the above-described hand presence range (see FIG. 6) is not estimated, it is required to perform processing for detecting the hand 220 on the entire image. In this case, the time until the hand 220 is detected and the load due to the detection process increase due to the wide processing range.

On the other hand, in the video processing apparatus according to the present embodiment, the process of detecting the hand 220 is performed only in the hand presence range in the image. Therefore, it is possible to detect a hand very efficiently. Furthermore, since the process of detecting the hand only in the hand presence range is performed, for example, when the hand of another person exists outside the hand presence range, the hand of the other person is detected. Can be prevented from being detected as a hand. That is, it is possible to prevent erroneous detection and accurately detect a human hand.

When the detection of the hand 220 is completed, the output unit 150 outputs the detected hand information as a result (step S05). The output unit 150 outputs, for example, the detected position, scale, and rotation angle of the hand 220. If a plurality of hands 220 are detected, the results are output for all hands 220.

As described above, according to the video processing apparatus according to the first embodiment, the hand presence range is estimated based on the detected face information. Therefore, it is possible to efficiently and accurately detect the hand of a person in the video.

<Second embodiment>
Next, a video processing apparatus according to the second embodiment will be described with reference to FIGS. FIG. 9 is a block diagram showing the configuration of the video processing apparatus according to the second embodiment. FIG. 10 is a flowchart showing the operation of the video processing apparatus according to the second embodiment. The second embodiment differs from the first embodiment described above in part of the configuration and operation, and the other portions are substantially the same. For this reason, in the following description, a different part from 1st Example is described in detail, and description is abbreviate | omitted suitably about another overlapping part.

In FIG. 9, the video processing apparatus according to the second embodiment is configured to include a skin color area detection unit 160, a difference area detection unit 170, and a memory 175 in addition to the same configuration as that of the first embodiment. Yes.

The skin color area detection unit 160 is an example of the “skin color area detection unit” of the present invention, and detects a skin color area in an image. The skin color region is detected based on, for example, a preset skin color component. Or you may detect according to the color of the detected face 210. FIG.

The difference area detection unit 170 is an example of the “difference area detection unit” of the present invention, and the difference in luminance between the past image (for example, the image one frame before) stored in the memory 175 and the current image is A difference area that is equal to or greater than a predetermined value is detected.

In FIG. 10, during the operation of the video processing apparatus according to the second embodiment, first, the video input unit 110 acquires an image constituting the video (step S21). When the image is acquired, the face detection unit 120 detects the face 210 of the person 200 in the image (step S22). That is, the same processing as in the first embodiment is performed.

Here, particularly in the second embodiment, the skin color area detection unit 160 detects a skin color area in the image (step S23). Further, the difference area detection unit 170 detects a difference area in the image using the past image stored in the memory 175 (step S24).

When the skin color area and the difference area are detected, as shown in FIG. 6, the circle P1 centered on the detected face 210 is estimated as the hand presence range (step S25). However, from the hand presence estimation range here, an area that is not a skin color area and an area that is not a difference area are excluded. That is, the hand presence range is within the circle P1 centering on the face 210, and is a skin color region and a difference region.

The color of the hand 220 is considered to be a skin color unless, for example, gloves are used. In addition, the hand 220 in the image is considered to move more than other objects. For this reason, there is a high possibility that the position of the hand 200 is different between the past image and the current image. Therefore, it is estimated that there is a high possibility that the hand 220 is present in an area where the luminance difference is equal to or greater than a predetermined value (that is, an area where the luminance has greatly changed). As a result, if the skin color area and the difference area are detected, the range in which the hand detection process is performed (that is, the hand presence range) can be further narrowed.

When the hand presence range is estimated, hand detection processing is performed in the hand detection unit 140 (step S26). Then, the output unit 150 outputs the detected hand information as a result (step S27).

As described above, according to the video processing apparatus according to the second embodiment, the hand presence range is estimated based on the skin color area and the difference area. Therefore, it is possible to detect the hand of a person in the video more efficiently and accurately.

<Third embodiment>
Next, a video processing apparatus according to the third embodiment will be described with reference to FIGS. FIG. 11 is a block diagram showing the configuration of the video processing apparatus according to the third embodiment. FIG. 12 is a conceptual diagram showing a plurality of images used for learning, and FIG. 13 is a plan view conceptually showing a hand presence range estimated based on learning. The third embodiment differs from the first and second embodiments described above in part of the configuration and operation, and the other portions are substantially the same. For this reason, in the following description, a different part from 1st and 2nd Example is described in detail, and description is abbreviate | omitted suitably about another overlapping part.

In FIG. 11, the video processing apparatus according to the third embodiment includes a learning unit 180 and a learning storage unit 185 in addition to the same configuration as that of the first embodiment.

The learning unit 180 is an example of the “learning unit” of the present invention, and learns the relative positional relationship between the face of the person and the hand from the image stored in the learning storage unit 185. The learning storage unit 185 stores a plurality of images, for example, as shown in FIG.

The learning in the learning unit 180 may be performed every time an image is acquired in the video input unit 110, or may be learned in advance using a predetermined image. That is, the learning unit 180 may perform learning in real time during operation of the apparatus, or may simply store the results of learning performed in advance.

13, the hand presence range estimation unit 130 according to the third example estimates the hand presence range based on the learning result in the learning unit 180, unlike the first example and the second example described above. Specifically, as shown in the figure, a range P2 in which the possibility of the hand 220 being present is high with respect to the position of the face 210. In addition, in FIG. 13, it has shown that the possibility of presence of a hand becomes high, so that it is an inner circle | round | yen. The hand detection process is performed in order from the inner circle (that is, the region where m hands are highly likely to exist).

If the hand presence range is estimated based on the learning result, as shown in FIG. 6, for example, a more appropriate hand presence range can be estimated as compared to the case where the circle P1 centered on the face 210 is estimated as the hand presence range. . Specifically, since the upper portion and the right portion of the face 210 in the figure having a low hand presence probability can be excluded from the hand presence range, the hand presence range can be estimated as a narrower range. Therefore, the efficiency of processing for detecting a hand can be further increased.

As described above, according to the video processing apparatus according to the third embodiment, the hand presence range is estimated based on the learning result. Therefore, it is possible to detect the hand of a person in the video more efficiently and accurately.

The present invention is not limited to the above-described embodiment, and can be appropriately changed without departing from the gist or concept of the invention that can be read from the claims and the entire specification. And methods are also within the scope of the present invention.

110 video input unit 120 face detection unit 130 hand presence range estimation unit 140 hand detection unit 150 output unit 160 skin color region detection unit 170 difference region detection unit 175 memory 180 learning unit 185 learning storage unit 200 person 210 face 220 hand

Claims (8)

Face detection means for detecting a human face from an image;
Range estimation means for estimating a hand presence range in which the person's hand may exist based on the detected face information of the person;
An image processing apparatus comprising: hand detection means for performing processing for detecting the hand of the person in the hand presence range. The video processing apparatus according to claim 1, wherein the range estimation means estimates a circle centered on the face of the person as the hand presence range. Learning means for learning the relative positional relationship of the person's face and the person's hand;
The video processing apparatus according to claim 1, wherein the range estimation unit estimates the hand presence range based on the learned relative positional relationship. The video processing apparatus according to claim 1, wherein the hand detection means includes scale estimation means for estimating a size of the person's hand based on information on the face of the person. The video processing apparatus according to claim 1, wherein the hand detection means includes rotation angle estimation means for estimating a rotation angle of the person's hand based on information on the face of the person. The hand detection means includes skin color area detection means for detecting a skin color area from the image, and detects the human hand in the skin color area in the hand presence range. The video processing apparatus according to claim 1 in the range. The hand detection means includes difference area detection means for comparing the past image and the current image, and detecting a difference area in which a difference in luminance is equal to or greater than a predetermined value. The video processing apparatus according to claim 1, wherein a hand of the person is detected in a difference area. A face detection process for detecting a human face from an image;
A range estimation step of estimating a hand presence range in which the person's hand may exist based on the detected face information of the person;
And a hand detecting step of performing a process of detecting the hand of the person in the hand presence range.

Images