JP6762544B2 - Image processing equipment, image processing method, and image processing program - Google Patents

Description

The present invention relates to an image processing device, an image processing method, and an image processing program for recognizing an image of an object in an image.

In recent years, technology that allows users to experience virtual reality (hereinafter also referred to as "VR") has been used in various fields such as games, entertainment, and vocational training. In VR, a glasses-type or goggle-type display device called a head-mounted display (hereinafter, also referred to as “HMD”) is usually used. By attaching the HMD to the user's head and viewing the screen built into the HMD with both eyes, the user can appreciate a three-dimensional image. The HMD has a built-in gyro sensor and an acceleration sensor, and the image displayed on the screen changes according to the movement of the user's head detected by these sensors. As a result, the user can experience as if he / she is in the displayed image.

In such a VR technical field, a user interface for performing operations such as switching images and moving objects in an image by a user's gesture is being studied. For example, in Patent Document 1, a user interface is arranged in a virtual space, an operation unit used by the user to operate the user interface is within the field of view of the imaging unit, and the positional relationship between the operation unit and the user interface. Discloses an image processing device that determines that there has been an operation on the user interface when is in a specified positional relationship.

Further, Patent Document 2 discloses a technique of recognizing an object included in a moving image by using a color feature amount. Specifically, a first reference color, which is a hue that occupies the largest area of the hue of the frame image to be processed, and an RGB histogram of the frame image to be processed are created, and a predetermined threshold value or more of the histogram is created. In addition, the second reference color showing the peak farthest from the first reference color is specified, and the closed region existing in the frame image is specified by edge detection, and the inside of the closed region is specified. Corresponds to a closed region that includes the first and second reference colors specified in the image frame processed immediately before the frame image and is recognized as the object in the previous image frame. A closed area that overlaps at least a part of the area is recognized as the object.

Japanese Unexamined Patent Publication No. 2012-48656 Japanese Patent No. 5887264

When performing an operation by a gesture, it is necessary to accurately recognize the image of the object to be gestured from the image. However, when performing a gesture with the user's hand or finger, the color of the hand or finger to be recognized varies greatly from person to person, and even if the hand or finger of the same person is used, the color differs depending on the part. It is very important to set an appropriate amount of color features in order to extract the recognition target in just proportion.

Regarding the color feature amount for object recognition, in Patent Document 2, from the closed region determined as the object to be recognized in the previous frame image, the color feature amount used in the processing of the next image frame (first). And a second reference color) (see paragraph 0046 of Patent Document 2). However, in the first place, Patent Document 2 does not disclose details on how to obtain an appropriate color feature amount for the frame image to be processed first.

The present invention has been made in view of the above, and is an image processing apparatus, an image processing method, and an image processing method capable of setting an appropriate color feature amount for accurately recognizing an image of a specific object in an image. An object of the present invention is to provide an image processing program.

In order to solve the above problems, the image processing device according to one aspect of the present invention is an image processing device that recognizes an image of the object from an image of a specific object, and is a space in which the image processing device exists. An image region of the object is extracted from the image by using an external information acquisition unit that acquires image information about the object and color features of a plurality of pixels at predetermined positions in the image based on the image information. Based on the specific point acquisition unit that acquires the position of the specific point in the region and the color feature amount of a plurality of pixels located within a predetermined range including the specific point, the color feature amount for recognizing the object is obtained. It is provided with a color feature amount setting unit for setting.

The image processing device further includes a display unit that displays the image on the screen and an image composition unit that synthesizes and displays a predetermined object on the screen on which the image is displayed, and the predetermined position is the said. It may be set inside the object.

In the image processing apparatus, the specific point acquisition unit creates a histogram of the color features of a plurality of pixels at the predetermined positions, and creates a region of pixels having the color features included in the predetermined range of the histogram. An object area extraction unit extracted from the image as a candidate area for the image of the object and a contour of the candidate region are extracted, and based on the contour, whether or not the candidate region is a region of the image of the object is determined. The object determination unit to be determined, and a specific point determination that outputs a specific point in the candidate area as a specific point in the image area of the object when it is determined that the candidate area is an image area of the object. It may have a part and.

In the image processing device, when the area of the candidate region is equal to or larger than a predetermined value, the object determination unit may determine that the candidate region is an image region of the object.

In the image processing apparatus, when a plurality of the candidate regions are extracted, the object determination unit may determine whether or not the area is equal to or larger than the predetermined value for the candidate region having the largest area.

In the image processing apparatus, the object is a user's hand, and the object determination unit detects irregularities in the contour of the candidate region, has four or more convex portions, and is each from the position of the center of gravity of the contour. When the average value of the distances to the convex portions is a predetermined times or more the average value of the distances from the specific points to the concave portions, it may be determined that the candidate region is the region of the image of the object.

In the image processing apparatus, the color feature amount setting unit calculates an average value, a median value, or a mode value of color feature amounts of a plurality of pixels located within the predetermined range including the specific point, and the average value. A value in a predetermined range including a value, a median value, or a mode value may be set as a color feature amount for recognizing the object.

In the image processing apparatus, the color feature amount may include at least hue in the HSV color space.

An image processing method according to another aspect of the present invention is an image processing method executed by an image processing device that recognizes an image of the object from an image of a specific object, and the image processing device is the image processing. It is provided with an external information acquisition unit that acquires image information about the space in which the device exists, and a region of an image of the object from the image is used by using color features of a plurality of pixels at predetermined positions in the image based on the image information. Is extracted, and the object is recognized based on the step (a) of acquiring the position of a specific point in the region of the image and the color features of a plurality of pixels located within a predetermined range including the specific point. It includes a step (b) of setting a color feature amount for the purpose.

An image processing program according to still another aspect of the present invention is an image processing program executed by an image processing device that recognizes an image of the object from an image of a specific object, and the image processing device is the image. An external information acquisition unit that acquires image information about the space in which the processing device exists is provided, and color features of a plurality of pixels at predetermined positions in the image based on the image information are used to obtain an image of the object from the image. The object is recognized based on the step (a) of extracting a region and acquiring the position of a specific point in the region of the image and the color features of a plurality of pixels located within a predetermined range including the specific point. The step (b) of setting the color feature amount for this purpose is executed.

According to the present invention, a region of an image of an object to be recognized is tentatively extracted from the image by using the color features of a plurality of pixels at predetermined positions in the image, and a specific point in the region is tentatively extracted. Since the color feature amount for recognizing the object is set based on the color feature amount of a plurality of pixels located around the image, an appropriate color for accurately recognizing the image of a specific object in the image. It is possible to set the feature amount.

It is a block diagram which shows the schematic structure of the image processing apparatus which concerns on embodiment of this invention. It is a schematic diagram which shows the state which the image processing apparatus was attached to the user. It is a schematic diagram which illustrates the screen displayed on the display part shown in FIG. It is a flowchart which shows the operation of the image processing apparatus shown in FIG. It is a schematic diagram which illustrates the screen displayed on the display part shown in FIG. It is a schematic diagram which illustrates the screen displayed on the display part shown in FIG. It is a flowchart which shows the acquisition process of the center of gravity position of a hand image. It is a schematic diagram for demonstrating the acquisition process of the center of gravity position of the hand image. It is a flowchart which shows the process of setting the range of a color feature amount based on the position of the center of gravity. It is a schematic diagram which shows the state which the image of the hand which is a recognition target is extracted.

Hereinafter, the display device according to the embodiment of the present invention will be described with reference to the drawings. The present invention is not limited to these embodiments. Further, in the description of each drawing, the same parts are indicated by the same reference numerals.

In the following embodiment, as an example, a case where the present invention is applied to a so-called stereoscopic image processing device provided with a screen for allowing a user to recognize a three-dimensional space will be described. However, the present invention is not limited to this, and can be applied to various image processing devices that recognize an image of a specific object in an image and perform various processes.

FIG. 1 is a block diagram showing a schematic configuration of an image processing apparatus according to an embodiment of the present invention. The image processing device 1 according to the present embodiment is a device that allows the user to recognize the three-dimensional space by viewing the screen with both eyes, and as shown in FIG. 1, the display unit 11 and the storage unit 12 A calculation unit 13 that performs various calculation processes, and an external information acquisition unit 14 that acquires information (hereinafter, referred to as external information) regarding the outside of the image processing device 1 are provided.

FIG. 2 is a schematic view showing a state in which the image processing device 1 is attached to the user 2. As the image processing device 1, a general-purpose device provided with a display and a camera, such as a smartphone, a personal digital assistant (PDA), and a portable game device, can be used. When a smartphone is used as the image processing device 1, for example, the user can view the display hands-free by attaching the display to the holder 3 with the front side provided with the display facing the user. Inside the holder 3, two lenses are attached at positions corresponding to the left and right eyes of the user. When displaying the content for stereoscopic viewing on the image processing device 1, the user can recognize the image displayed on the display as if it were a three-dimensional space by looking at the display through these lenses. In addition, a camera 4 is generally provided on the back surface of the smartphone, and the camera 4 captures the space around the user 2 to capture an image, and the image is displayed as it is on the display, or the image is prepared in advance. It can be combined with the content (video or still image) and displayed.

However, the appearance of the image processing device 1 is not limited to that shown in FIG. For example, an image processing device dedicated to stereoscopic viewing in which an image processing device and a holder are integrated may be used. Such a dedicated image processing device is also called a head-mounted display. Further, the image processing device 1 is not limited to a small portable device such as a smartphone, and may be used, for example, by connecting a webcam to a desktop personal computer, a notebook personal computer having a built-in camera, or the like. A tablet terminal may be used.

Referring to FIG. 1 again, the display unit 11 is a display including a display panel and a drive unit formed of, for example, a liquid crystal or an organic EL (electroluminescence). FIG. 3 is a schematic diagram showing an example of a screen displayed on the display unit 11. When the user is made to recognize a still image or a moving image three-dimensionally, as shown in FIG. 3, the display panel of the display unit 11 is divided into two areas, and two images having parallax with each other are divided into these areas 11a. , 11b. The user 2 can recognize the three-dimensional space by viewing the two images displayed in the areas 11a and 11b with the left and right eyes, respectively.

The storage unit 12 is a computer-readable storage medium such as a semiconductor memory such as a ROM or RAM. In addition to the operating system program and the driver program, the storage unit 12 is an image of an application program that executes various functions, a program storage unit 121 that stores various parameters used during execution of these programs, and the like. It has a color feature amount storage unit 122 that stores a color feature amount used in image processing for recognizing an image of a specific object. In addition, the storage unit 12 may store image data and audio data of various contents displayed on the display unit 11, image data of the image acquired by the camera 4, and the like.

The arithmetic unit 13 is configured by using, for example, a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit), and by reading various programs stored in the program storage unit 121, each unit of the image processing device 1 is integrated. While controlling it, it executes various arithmetic processes for displaying various images. The detailed configuration of the calculation unit 13 will be described later.

The external information acquisition unit 14 acquires image information (image data) related to the real space around the image processing device 1. The configuration of the external information acquisition unit 14 is not particularly limited as long as it can detect the position and movement of surrounding objects. For example, an optical camera, an infrared camera, an ultrasonic transmitter, a receiver, and the like are included in the external information acquisition unit 14. Can be used as. For example, when a smartphone as shown in FIG. 2 is used as the image processing device 1, the camera 4 built in the smartphone is used as the external information acquisition unit 14.

Next, the detailed configuration of the calculation unit 13 will be described. By reading the image processing program stored in the program storage unit 121, the calculation unit 13 generates an image based on the image information acquired by the external information acquisition unit 14, and also generates an image of a specific object captured in the image. Performs image processing to recognize the image.

As shown in FIG. 1, the calculation unit 13 acquires the position of the center of gravity for setting a color feature amount (hereinafter, also referred to as a color feature amount for object recognition) used when recognizing an image of a specific object captured in an image. The image displayed on the display unit 11 is combined with the unit 131 and the color feature setting unit 132, the object recognition unit 133 that recognizes the image of a specific object in the image using the set color feature amount, and the display unit 11. It has an image compositing unit 134. In other words, the center of gravity position acquisition unit 131 and the color feature amount setting unit 132 calibrate the color feature amount used in the object recognition unit 133.

The center of gravity position acquisition unit 131 is a specific point acquisition unit that extracts a region of an image of a specific object from an image and acquires a specific point in the region. In the present embodiment, the position of the center of gravity of the extracted region is acquired as a specific point. Specifically, the center of gravity position acquisition unit 131 includes an object region extraction unit 131a, an object determination unit 131b, and a center of gravity determination unit 131c. Among them, the object region extraction unit 131a extracts a candidate region of an image of a specific object from the image based on the color features of a plurality of pixels at predetermined positions in the image. The object determination unit 131b determines whether or not the candidate region extracted by the object region extraction unit 131a is an image region of a specific object. The center of gravity determination unit 131c is a specific point determination unit that outputs the position of the center of gravity of the candidate region determined to be the region of the specific image as a specific point.

The color feature amount setting unit 132 sets the color feature amount for object recognition based on the color feature amount of a plurality of pixels at the specific point (center of gravity position) and its vicinity. Specifically, the color feature amount setting unit 132 calculates an average value of color feature amounts of a plurality of pixels located within a predetermined range including a specific point (center of gravity position), and sets a value in a predetermined range including the average value. It is set as a color feature amount for object recognition.

The object recognition unit 133 recognizes an image of a specific object in the image by extracting pixels having the color feature amount for object recognition set by the color feature amount setting unit 132 from the image.

The image synthesizing unit 134 synthesizes an object used when setting a color feature amount for object recognition with an image in the real space, or prepares an image of the object recognized by the object recognizing unit 133 in advance. Performs image processing such as compositing with the image of.

Next, the operation of the image processing device 1 will be described.
In the present embodiment, the object to be recognized is not particularly limited, and various objects such as a user's hand, finger, and stylus pen can be recognized. However, in order to set the color feature amount for object recognition, the general shape feature of the object to be recognized is stored in the storage unit 12 in advance. For example, a human hand usually has a convex portion corresponding to a fingertip and a concave portion corresponding to between fingers, and from one point (for example, the center of gravity) in the palm to each convex portion. The feature is that the distance is longer than the distance from the same point to each recess. In this case, the characteristics of the general shape of a human hand are that the contour has at least four convex portions, and the average value of the distance from the center of gravity position to each convex portion is the average of the distances from the center of gravity position to each concave portion. It can be expressed as a predetermined multiple of the value. Therefore, such a feature is stored in the storage unit 12. In the following, the user's hand will be described as a recognition target.

The type of color feature amount for object recognition is not particularly limited, and various values such as pixel value (RGB value), color ratio, color difference, hue in HSV color space, lightness, and saturation may be used. Can be done. In the following, a case where each value obtained by converting a pixel value (RGB value) into hue, saturation, and lightness in the HSV color space is used as a color feature amount will be described.

Here, when RGB values are used, it is necessary to express the hue with three values, whereas when using the hue in the HSV color space, there is an advantage that the hue can be expressed with one value. In addition, when hue is used, it is possible to suppress the influence of fluctuations in the brightness of the image and differences in the way light hits the object to be recognized.

FIG. 4 is a flowchart showing the operation of the image processing device 1. 5 and 6 are schematic views illustrating the screen displayed on the display unit 11, and the image of the external space acquired by the external information acquisition unit 14 is displayed in the two areas 11a and 11b in the screen. Indicates the state of being.

First, in step S10, the image synthesizing unit 134 draws a hand-shaped frame m1 on the screen (see FIG. 5). Specifically, the object in the shape of a hand is combined with the image of the external space displayed in each of the areas 11a and 11b in the screen and displayed. At this time, the image synthesizing unit 134 may display a message m2 instructing the frame m1 to put the user's hand on the screen (each area 11a, 11b). The user adjusts the position of his / her hand with respect to the external information acquisition unit 14 (for example, the camera 4 shown in FIG. 2), and as shown in FIG. 6, the image of the hand reflected on the screen (each area 11a, 11b). Align m4 with the frame m1.

In the following step S12, the center-of-gravity position acquisition unit 131 acquires the position of the center of gravity of the hand image m4 captured on the screen (each region 11a, 11b). FIG. 7 is a flowchart showing a process of acquiring the position of the center of gravity of the hand image. FIG. 8 is a schematic diagram for explaining the process of acquiring the position of the center of gravity of the hand image.

First, in step S120, the center of gravity position acquisition unit 131 acquires an image of the real space captured by the external information acquisition unit 14, and temporarily stores it in the storage unit 12.

In the following step S121, the object area extraction unit 131a acquires the pixel value (RGB value) of the pixel located in the area m3 of the predetermined coordinates in the preset image. Here, since the area m3 is set so as to fit inside the hand-shaped frame m1, when the user adjusts the image m4 of his / her hand to the frame m1, in most cases, at least the inside of the area m3 is the user. It will be occupied by the hand image m4.

In the following step S122, the object area extraction unit 131a converts the pixel values (RGB values) of the plurality of pixels located in the area m3 into the color feature amounts (that is, hue, saturation, brightness) in the HSV color space. Convert.

In the following step S123, the object area extraction unit 131a creates a histogram of the color features of a plurality of pixels located in the area m3. Specifically, a histogram is created for each of the hue, saturation, and lightness converted in step S122.

In the following step S124, the object area extraction unit 131a sets the range of the color feature amount for extracting the hand image m4 based on the histogram created in step S123. Specifically, in the histogram of each color feature amount (hue, saturation, lightness), a predetermined range including the maximum frequency color feature amount is set as a range of the color feature amount for extracting the hand image m4. For example, the range of hue, saturation, and lightness included in the first to predetermined frequency classes is set. Alternatively, a range such as the mode of hue ± Δ ° and the mode of saturation or lightness ± Δ% may be set (the symbol Δ is an arbitrary value).

In the following step S125, the object area extraction unit 131a extracts the pixel having the color feature amount set in step S124 from the entire image. Specifically, the binarization process is executed with both ends of the range of the color feature amount determined in step S124 as threshold values. As a result, a region in the area m3 shown in FIG. 6 that approximates the typical color feature amount of the pixel is extracted from the image. Such processing is also called histogram backprojection.

In the following step S126, the object determination unit 131b extracts the contour by tracking the contour of the region extracted in step S125. Specifically, assuming that the pixel having a value in the binary image (the pixel extracted in step S125) is a black pixel and the other pixels are white pixels, first, a pixel point that changes from a white pixel to a black pixel by raster scanning. To explore. When a black pixel is found, the black pixel is searched by turning clockwise around the black pixel starting from the searched direction (approach direction). Subsequently, the black pixel is moved to the found black pixel, and the black pixel is searched for by turning clockwise around the black pixel starting from the approach direction. This process is repeated, and if the found black pixel is the start point (the first black pixel found) and the next moving point has already been searched, the process ends.

By the processing up to this point, as shown in FIG. 8, a region close to the image of the hand (see contour m5) can be extracted, but at the same time, a region close to the color of the image of the hand (see contours m6 and m7) can be extracted. May be done. Hereinafter, the area surrounded by the extracted contour is referred to as a candidate area.

In step S127, when a plurality of candidate regions are extracted, the object determination unit 131b selects the candidate region having the largest area. Here, the color feature amount used in step S125 is set based on the color feature amount of the hand image m4 (see FIG. 6) superimposed on the area m3, and therefore is usually as shown in FIG. This is because the contour m5 of the region included in the hand image m4 is larger than the contours m6 and m7 of the region included in the image of another object.

In the following step S128, the object determination unit 131b determines whether or not the area of the selected candidate region is equal to or larger than a predetermined value. Here, since the hand image m4 is arranged so as to fit the hand-shaped frame m1, if the candidate area selected in step S127 is included in the hand image m4, the size is larger than a certain size. Because it should be. By discriminating by area in this way, even if an image of another person's hand with similar colors is reflected in the image, the color feature amount for object recognition is set based on the image of the other person. It is possible to prevent the situation where it ends up. The predetermined value used in this determination is appropriately set based on the size of the hand-shaped frame m1.

When the area of the selected candidate region is equal to or larger than a predetermined value (step S128: Yes), the object determination unit 131b determines the unevenness of the contour of the candidate region (step S129). Specifically, corner points are detected from the contour of the candidate region, and it is determined whether each corner point is a concave portion or a convex portion. As a method for detecting a corner point, various known methods can be used. As an example, a P _i arbitrary point on the contour, the two points P _ik apart back and forth along the k pixels outline only against the point P _i, between a line segment connecting each and P i _{+ k} When the angle formed is smaller than 180 ° or larger than 180 °, the point P _i is detected as a square point. Then, when the angle formed by the line segment P _ik P _i and the line segment P _i P _{i + k} at the detected corner point is an acute angle, the corner point is determined to be a convex portion. If the angle exceeds 180 °, it is determined that the angle is a recess. Strictly speaking, an angle point having an angle greater than 90 ° and less than 180 ° (that is, an obtuse angle) is a convex portion, but in step S129, the obtuse angle is intended to detect an area corresponding to the user's finger. The corner points are not regarded as convex parts. In the contour m5 shown in FIG. 8, the corner points P1, P2, P3, P4, and P5 are determined to be convex portions, and the corner points Q1, Q2, Q3, and Q4 are determined to be concave portions.

In the following step S130, the object determination unit 131b determines whether or not there are four or more convex portions in the contour of the determination target. The reason why the number of fingers is four or more is that it is possible that all five fingers cannot be detected as convex parts when the fingers are close to each other.

When four or more convex portions are present (step S130: Yes), the object determination unit 131b calculates the position of the center of gravity (center of gravity point G) of the candidate region (step S131).

In the following step S132, the object determination unit 131b calculates the distance from the position of the center of gravity to each concave portion and each convex portion. For example, in FIG. 8, the distances between the center of gravity point G and the angle points P1, P2, P3, P4, P5, Q1, Q2, Q3, and Q4 are calculated, respectively.

In the following step S133, the object determination unit 131b determines whether or not the average value (average distance) of the distance from the center of gravity position to the convex portion is a predetermined times or more the average value (same as above) of the distance from the center of gravity position to the concave portion. To judge. Here, in a human hand, the distance from the center of gravity position at the approximate center of the palm to the fingertip is longer than the distance from the center of gravity position to the end of the palm, and although there are individual differences, it is generally at least 1.2. It can be said that it is more than double. Therefore, as shown in FIG. 8, the average distance from the center of gravity point G to the convex portions P1, P2, P3, P4, and P5 and the concave corner points Q1, Q2, Q3, and Q4 from the center of gravity point G The average distance to is calculated, and based on the ratio of both, it is determined whether or not the shape of the extracted candidate region is close to the shape of the hand.

When the average distance from the center of gravity position to the convex portion is equal to or more than a predetermined time of the average distance from the center of gravity position to the concave portion (step S133: Yes), in the object determination unit 131b, the selected candidate region is the region of the hand image. Judge that there is. In this case, the center of gravity determination unit 131c determines the position of the center of gravity of the candidate region as the position of the center of gravity of the hand image and outputs it (step S134). After that, the process returns to the main routine.

On the other hand, when the area of the candidate region selected in step S127 is less than a predetermined value (step S128: No), and when the contour of the selected candidate region has three or less convex portions (step S130: No). Alternatively, when the average distance from the center of gravity position to the convex portion is less than a predetermined time of the average distance from the center of gravity position to the concave portion (step S133: No), the process returns to step S120.

Referring to FIG. 4 again, in step S14 following step S12, the color feature amount setting unit 132 sets the color feature amount for object recognition based on the position of the center of gravity acquired in step S12. FIG. 9 is a flowchart showing a process of setting a color feature amount for object recognition based on the position of the center of gravity.

First, in step S140, the color feature amount setting unit 132 determines the pixel values (RGB values) of the center of gravity position and the plurality of pixels in the vicinity thereof acquired in step S12 from the image acquired in step S120 (see FIG. 7). To get. The range of pixels for which the pixel value is acquired is set in advance, for example, within the range of several pixels to several tens of pixels from the position of the center of gravity.

In the following step S141, the color feature amount setting unit 132 converts each pixel value (RGB value) of the plurality of pixels acquired in step S140 into a color feature amount (hue, saturation, lightness) in the HSV color space. To do.

In the following step S142, the color feature amount setting unit 132 calculates the average value of the color feature amounts among the plurality of pixels acquired in step S140. Specifically, the average value is calculated for each of hue, saturation, and lightness.

In the following step S143, the color feature amount setting unit 132 sets the color feature amount for object recognition based on the average value calculated in step S142. Specifically, for each of hue, saturation, and lightness, a predetermined range including an average value is set as a color feature amount for object recognition. For example, a range such as an average value of hue ± Δ ° and an average value of saturation or lightness ± Δ% is set (the symbol Δ is an arbitrary value). After that, the process returns to the main routine.

In step S16 of FIG. 4, the object recognition unit 133 extracts pixels from the image acquired by the external information acquisition unit 14 by using the color feature amount for object recognition set in step S14. The pixel area extracted in this way is recognized as a hand image. FIG. 10 is a schematic view showing a state in which the image m8 of the hand to be recognized is extracted.

As described above, according to the present embodiment, a candidate for a hand image is used by using a color feature amount tentatively set based on the pixel values of the pixels in the area m3 set in advance in the image. When an area is extracted and it is determined that this candidate area has hand-shaped features, the color feature amount for object recognition is set based on the position of the center of gravity of the candidate area and the pixel values of the pixels in the vicinity thereof. Therefore, by using the color feature amount for object recognition set in this way, it is possible to accurately recognize the image of the hand in the image. In particular, as in the present embodiment, even when the user's hand, which has a large individual difference or a difference between places with respect to the color feature amount, is to be recognized, the color feature amount for object recognition suitable for the user's hand can be obtained. It can be set with high accuracy.

In the above embodiment, as a specific point, the position of the center of gravity of the region of the image of the object to be recognized extracted from the image is acquired, but the specific point is not limited to this. For example, a point where the horizontal center line and the vertical center line of the image region of the object to be recognized intersect, a corner point, or a diagonal intersection connecting the corner points may be set as a specific point. .. As the specific point, a portion where a typical color of the object appears may be appropriately set according to the object to be recognized.

Further, in the above embodiment, the hue, saturation, and lightness in the HSV color space are used as the color feature amount, but other values may be used. For example, when using the RGB values of pixels, when acquiring the position of the center of gravity of the hand image (see FIG. 7), step S122 is omitted, and histograms are created for each of the R value, G value, and B value (step S123). ), Each range of R value, G value, and B value is set based on these histograms (step S124), and pixels having R value, G value, and B value within the set range may be extracted (step S124). Step S125). Further, when setting the color feature amount for object recognition based on the position of the center of gravity (see FIG. 9), step S141 is omitted and the average value is calculated for each of the R value, G value, and B value (step). S142), the range of the R value, the G value, and the B value may be set based on these average values (step S143). Alternatively, as the color feature amount, only the hue in the HSV color space may be used, or a combination such as hue and saturation, or hue and lightness may be used.

Further, in the above embodiment, the average value of the color features of the pixels at the specific point of the image of the object to be recognized extracted from the image and the pixels in the vicinity thereof is calculated (step S142), and the object is based on this average value. Although the color feature amount for recognition is set (step S143), it is not limited to the average value, but is used for object recognition based on statistical values such as the median value and the mode value of the color feature amount of pixels at a specific point and its vicinity. The color feature amount of may be set.

The present invention is not limited to the above-described embodiments and modifications, and various inventions can be formed by appropriately combining a plurality of components disclosed in the above-described embodiments and modifications. For example, some components may be excluded from all the components shown in the above-described embodiment and the modified example, or the components shown in the above-described embodiment and the modified example may be appropriately combined and formed. good.

1 Image processing device 2 User 3 Holder 4 Camera 11 Display unit 11a, 11b Area 12 Storage unit 13 Calculation unit 14 External information acquisition unit 121 Program storage unit 122 Color feature amount storage unit 131 Center of gravity position acquisition unit 131a Object area extraction unit 131b Object Judgment unit 131c Center of gravity determination unit 132 Color feature amount setting unit 133 Object recognition unit 134 Image composition unit

Claims (9)

An image processing device that recognizes an image of a specific object from an image of the object.
An external information acquisition unit that acquires image information about the space in which the image processing device exists, and
Using the color features of a plurality of pixels at predetermined positions in an image based on the image information, a region of an image of the object is extracted from the image, and the position of a specific point in the region of the image is acquired. Point acquisition department and
A color feature amount setting unit that sets a color feature amount for recognizing the object based on the color feature amounts of a plurality of pixels located within a predetermined range including the specific point.
A display unit that displays the image on the screen and
An image compositing unit that synthesizes and displays a predetermined object on the screen on which the image is displayed,
With
An image processing device whose predetermined position is set inside the object. The specific point acquisition unit
A histogram of the color features of a plurality of pixels at the predetermined positions is created, and a region of pixels having color features included in the predetermined range of the histogram is extracted from the image as a candidate region for an image of the object. Object area extraction unit and
An object determination unit that extracts the contour of the candidate region and determines whether or not the candidate region is an image region of the object based on the contour.
When the candidate region is determined to be the region of the image of the object, the center of gravity determination unit that outputs a specific point in the candidate region as a specific point in the region of the image of the object.
The image processing apparatus according to claim 1. The image processing apparatus according to claim 3, wherein the object determination unit determines that the candidate region is an image region of the object when the area of the candidate region is equal to or larger than a predetermined value. The image processing apparatus according to claim 4, wherein the object determination unit determines whether or not the area of the candidate region having the largest area is equal to or greater than the predetermined value when a plurality of the candidate regions are extracted. The object is the user's hand
The object determination unit detects unevenness in the contour of the candidate region, has four or more convex portions, and the average value of the distances from the center of gravity position of the contour to each convex portion is each from the specific point. When the average value of the distances to the recesses is at least a predetermined time, it is determined that the candidate region is the region of the image of the object.
The image processing apparatus according to any one of claims 3 to 5. The color feature amount setting unit calculates an average value, a median value, or a mode value of the color feature amounts of a plurality of pixels located within the predetermined range including the specific point, and the average value, the median value, or the mode value. The image processing apparatus according to any one of claims 1, 3 to 6, wherein a value in a predetermined range including the mode value is set as a color feature amount for recognizing the object. The image processing apparatus according to any one of claims 1, 3 to 7, wherein the color feature amount includes at least a hue in the HSV color space. An image processing method executed by an image processing device that recognizes an image of a specific object from an image of the object.
The image processing device includes an external information acquisition unit that acquires image information about the space in which the image processing device exists, and a display unit that displays the image on a screen.
A step of extracting an image region of the object from the image and acquiring the position of a specific point in the image region using the color features of a plurality of pixels at predetermined positions in the image based on the image information. (A) and
The step (b) of setting the color feature amount for recognizing the object based on the color feature amount of a plurality of pixels located within a predetermined range including the specific point.
The step (c) of synthesizing and displaying a predetermined object on the screen on which the image is displayed, and
Including
An image processing method in which the predetermined position is set inside the object. An image processing program executed by an image processing device that recognizes an image of a specific object from an image of the object.
The image processing device includes an external information acquisition unit that acquires image information about the space in which the image processing device exists, and a display unit that displays the image on a screen.
A step of extracting an image region of the object from the image and acquiring the position of a specific point in the image region using the color features of a plurality of pixels at predetermined positions in the image based on the image information. (A) and
The step (b) of setting the color feature amount for recognizing the object based on the color feature amount of a plurality of pixels located within a predetermined range including the specific point.
The step (c) of synthesizing and displaying a predetermined object on the screen on which the image is displayed, and
To execute,
An image processing program in which the predetermined position is set inside the object.

Images