JP2010238134A - Image processor and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To generate a large number of images in a virtual space based on a small amount of data acquired in a real space and construct recognition database for recognizing a person in the images. <P>SOLUTION: An image processor acquires three-dimensional shape data of a person, applies the data to a joint model of a person; creates a human body model in the virtual space; moves each joint of a human body model corresponding to the movement of each joint of the joint model; generates individual differences of action and physical constitution by adding noise according to the differences in the virtual space to the human body model by a noise generation processing section 11; generates an image from the human-body model, based on a set photographing condition by an image generation processing section 13; calculates image features of a person in the image from the generated image by an image feature calculation processing section 14; and constructs a recognition database 54 for recognizing a person in the image by machine-learning the image features by a machine learning section 15. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to an image processing apparatus and a program for constructing an image recognition system by projecting a few human shape data acquired in a real space onto a three-dimensional virtual space.

  In general, an image recognition system that recognizes people in an image constructs a recognition database necessary for recognition by acquiring and analyzing a large number of images obtained by capturing the shape, posture, and movement of people. Recognize human movements and postures in real space images using data. At that time, the more types and the number of images to be analyzed, the higher the accuracy of the recognition database and the more accurately the shape, posture, and movement of a person can be recognized. In reality, however, the human body shape and movement, the surrounding environment and lighting, or the means and direction of photographing are various, and all images corresponding to the difference (noise) are photographed and acquired. Is extremely difficult.

  On the other hand, it is conceivable to obtain images of a large number of patterns by generating a desired image by changing a three-dimensional model of a person in a virtual space. Further, as such an apparatus, there is conventionally known an apparatus that generates a desired three-dimensional model by pasting a two-dimensional model image on a three-dimensional basic model (see Patent Document 1).

  However, in this conventional apparatus, the posture of the three-dimensional model is simply changed by moving a specific part of the two-dimensional model, and various images corresponding to the noise cannot be generated. And the difference tends to increase. As a result, an accurate image cannot be acquired, and the pattern or the like of the image that can be acquired is limited, and it is difficult to greatly increase the number of images.

Japanese Patent Laid-Open No. 11-175765

  The present invention has been made in view of such a conventional problem, and an object of the present invention is to easily obtain an image that is difficult to acquire in the real space in the virtual space based on a small amount of data acquired in the real space. In addition, an enormous number of images are generated, and images that are comparable to images acquired in real space are acquired. Another object is to construct a highly versatile recognition database using only data acquired in a virtual space by calculating predetermined image features from these images and performing machine learning.

The invention of claim 1 is an image processing device, means for acquiring human three-dimensional shape data, and means for applying a three-dimensional shape data to a human joint model in a virtual space to create a human body model in the virtual space A means for moving the human body model according to the movement of each joint of the joint model to change the human body model, a means for adding noise to the human body model according to the difference in the real space, and an image from the human body model. It is characterized by comprising means for generating and means for storing an image generated from a human body model.
According to a second aspect of the present invention, there is provided the image processing apparatus according to the first aspect, wherein the joint model is based on the displacement information for each joint of the joint model according to the movement and posture of the person, and the displacement information for each joint. And means for changing.
According to a third aspect of the present invention, in the image processing apparatus according to the first or second aspect, the image processing apparatus includes means for setting a photographing condition of an image acquired from the human body model, and the means for generating the image from the human body model is based on the human body model. An image based on shooting conditions is generated.
According to a fourth aspect of the present invention, in the image processing apparatus according to any one of the first to third aspects, the means for adding noise adds at least one noise of a human posture, motion, and physique to the human body model. It has the means to produce | generate an individual difference, It is characterized by the above-mentioned.
According to a fifth aspect of the present invention, in the image processing apparatus according to any one of the first to fourth aspects, the means for adding noise includes means for adding at least one noise of clothes, lighting, and shielding to a human body model. It is characterized by having.
According to a sixth aspect of the present invention, in the image processing apparatus according to any one of the first to fifth aspects, the means for calculating the human image feature in the image based on the image generated from the human body model, Means for constructing a recognition database for learning and recognizing a person in the image.
The invention according to claim 7 is a program for realizing each means of the image processing apparatus according to any one of claims 1 to 6 by a computer.

  According to the present invention, it is possible to easily generate an enormous number of images that are difficult to acquire in the real space in the virtual space, based on a small amount of data acquired in the real space. Moreover, since these images are added with noise corresponding to differences in the real space set in advance with respect to the human shape, posture, movement, lighting, etc., they are comparable to images acquired in the real space. Images can be acquired. By calculating predetermined image features from these images and performing machine learning, a highly versatile recognition database can be constructed using only data acquired in the virtual space.

1 is a block diagram schematically illustrating a configuration of an image processing apparatus according to an embodiment. It is a schematic diagram which shows a human joint model. It is a figure which shows the brightness | luminance pattern which produces | generates the difference in the pattern of clothes. It is a flowchart which shows the procedure of the image processing by the image processing apparatus of this embodiment.

An image processing apparatus according to an embodiment of the present invention will be described with reference to the drawings.
FIG. 1 is a block diagram schematically showing the configuration of the image processing apparatus of the present embodiment.

  The image processing apparatus 1 includes, for example, a central processing unit (CPU), a ROM (Read Only Memory) that stores various programs, and a RAM (Random Access Memory) that temporarily stores processing data for the CPU. Having a computer. The image processing apparatus 1 also includes an image processing unit 10, a noise generation function group 20 and an image feature calculation function group 40 each of which is stored in advance, and an image memory 30 that stores an image. Furthermore, the image processing apparatus 1 includes a human three-dimensional shape database 50 and a motion capture database 51, in which various data used for image processing are stored in advance.

  In this image processing apparatus 1, first, an image (distance image) obtained by photographing a person in a predetermined posture from front and rear, left and right, up and down using a distance image sensor such as a stereo camera, and viewed from all directions of the person. To get. Based on the acquired distance data of each image, for example, the surface of a person is measured three-dimensionally at a predetermined vertical and horizontal pitch, and the entire three-dimensional surface shape of the person is calculated as polygon data connecting measurement points. The shape data is acquired, and the acquired human shape data is stored in the three-dimensional shape database 50. In addition, the image processing apparatus 1 applies the three-dimensional shape data of the person created and acquired in this way to the joint model of the person in the virtual space to create a human body model in the virtual space.

FIG. 2 is a schematic diagram showing a human joint model.
The joint model 70 is a model that represents a person by connecting a plurality of joints. As shown in the figure, twelve joints 71, 72, 73 connected in order according to the position of each joint of the person, hands and feet. It consists of four end effectors 74 located at the tip. The joints 71, 72, and 73 are set such that the ten joints 71 and 72 can rotate and the positions of the four joints 72 in the joint model 70 are fixed.

Here, the motion capture database 51 acquires and stores data of a three-dimensional movement and posture of a person by a motion capture system that digitally stores the movement and posture of the person. That is, for example, a person with a marker is caused to perform a predetermined motion, the marker is continuously detected by a tracker, and one or a plurality of time-series data for each motion and posture is acquired and stored. Thereby, the change for each part of the joint model 70 is acquired, and the rotation angle, position, and displacement for each joint 71, 72, 73 of the joint model 70 according to the movement and posture of the person (herein, these are collectively referred to) (Referred to as displacement information) is sequentially stored as time-sequential data to construct a database.
The distance image of the person applied to the joint model 70 is desirably acquired as a distance image of the person in a posture that matches the posture of the joint model 70. Further, the joint model 70 is an example, and can be applied to any joint model.

  The image processing apparatus 1 displays an image (human body model) based on the above-described human three-dimensional shape data on a display means (not shown). The user designates positions corresponding to the joints 71, 72, 73 and the end effector 74 of the joint model 70 with respect to the displayed image of the human body model by operating the mouse or the like, so that the three-dimensional The image of the human body model representing the shape data and the positions of the joint model 70 are associated with each other. Accordingly, the image processing apparatus 1 applies the three-dimensional shape data of the human body model to the joint model 70, moves each joint of the human body model created in accordance with the movement of each joint of the joint model 70, and converts the human body model into the joint model 70. Change to match. In this way, the human body model and the positions and directions of the shapes of the respective parts are changed to create a human body model and three-dimensional shape data of a predetermined posture and motion.

  The image processing apparatus 1 reads out from the motion capture database 51 as described above, acquires displacement information for each joint 71, 72, 73 of the joint model 70 according to a specific motion and posture of a person, and based on the displacement information. The joint model 70 is changed by moving the joints 71, 72, and 73 by changing the rotation angles and positions thereof. At the same time, the noise generation processing unit 11 of the image processing unit 10 changes the three-dimensional shape data (human body model) of the person, and from a predetermined and small amount of the three-dimensional shape data of the person, various movements of the person depending on the human body model. To reproduce.

  That is, in the real space, there are various physique differences for each person, such as a thin person, a fat person, a tall person, or a short person. In addition, the shape changes variously depending on whether the person is wearing thick clothes or wearing lightly, and the person's movements are also different from person to person such as a slow-moving person or a fast-moving person. Therefore, the image processing apparatus 1 adds noise to the difference in the real space such as a difference in the physique, shape, and movement of the person in the real space, a difference in lighting and the environment, and the like by the noise generation processing unit 11. To reproduce. Based on the noise models 21 to 26 set in the noise generation function group 20, the noise generation processing unit 11 adds noise according to differences in the real space, such as the shape, movement, physique, and lighting of the person. The three-dimensional shape data and the human body model are changed. Hereinafter, although an example of various noise additions by the noise generation processing unit 11 will be described, these are examples and can be extended to any noise model for faithfully reproducing the real space.

  The noise generation processing unit 11 adds noise of the human physique to the human body model (three-dimensional shape data) based on the noise model 23 of the human physique and the noise model 24 of the human shape, and the individual difference of the physique Is generated. Specifically, the volume of the head, torso, arms, and feet of the human body model, the height, the length of the head, torso, arms, feet, and shoulder width, and the head, chest, abdomen, and buttocks・ Change the perimeter of the arms and feet. These values are centered on the Japanese standard size. And the upper limit and lower limit of each value are set arbitrarily. The difference between the upper limit value and the median value and the difference between the lower limit value and the median value are set to be the same. The appearance probability of each value follows a normal distribution in which an upper limit value and a lower limit value correspond to ± 3σ. (At this time, each value is changed each time by a predetermined amount, for example, 0.1 σ or the like. The same applies hereinafter). Here, the standard size of Japanese is applied to each value, but the standard size of the world, or the standard size of each country or region may be applied. In addition, a normal distribution is assumed for the probability density function of the variation of each value, but this can be replaced with statistical data of human body physique differences that have been medically elucidated using knowledge in the field of orthopedic medicine. it can.

  The noise generation processing unit 11 adds clothes noise such as clothes to the human body model based on the other noise model 24 to generate a difference in clothes. This difference in clothes includes a difference in thickness such as thin clothes and thick clothes, and a difference in clothes pattern. The difference in clothes pattern is the brightness generated by projecting a human body model of 3D shape data onto the image plane. It is sufficient that a sufficiently high correlation value can be obtained when the template matching process is performed on the image. Here, first, a predetermined luminance pattern 80 (see FIG. 3) is arranged and projected in a lattice pattern in the range where the clothes on the surface of the human body model are worn, and white noise in the range of ± 33% of the luminance gradation is applied to the luminance pattern 80. By adding, the difference in the pattern of the clothes is generated. That is, as the luminance gradation changes, for example, the black portion 81 of the luminance pattern 80 is not changed, and the white portion 82 is gradually blackened, or the white portion 82 is not changed and the black portion 81 is gradually whitened. The color of the pattern is gradually changed from white to black to generate a pattern difference. On the other hand, the difference in the shape and thickness of the clothes is generated in the same manner as the generation of the physique difference. However, as for the thickness of the clothing, the torso and arms have the largest fluctuations, and the head and feet are small.According to this rule, the circumference of the head, chest, abdomen, and buttocks, and the height, shoulder width, and torso Appropriate noise is added to the lengths of the head, arms, and feet to change the three-dimensional shape data, thereby generating a difference in the thickness of the human body model clothes.

  The noise generation processing unit 11 adds posture noise to the human body model based on the human posture noise model 22 to generate individual differences in posture. At that time, for each of the joints 71, 72, 73 of the joint model 70 (see FIG. 2), a plurality of joint angles with the same posture are acquired from the motion capture database 51 to obtain an average value and a standard deviation. The joint angle appearance probability follows a normal distribution, and noise is added to each joint angle within a range of ± 3σ with respect to the average value to change the three-dimensional shape data, and to generate individual differences in posture to create a human body model create. However, when a sufficiently large number of samples of the same posture can be acquired from the motion capture database 51, the variation of the optimum joint angle obtained by statistically analyzing the variation of each joint angle for each posture can be applied. . In addition, using the knowledge of rehabilitation medicine, a range of variation in the joint angle may be set with reference to the range of motion of each joint and physical load data applied to the joint for each posture.

  The noise generation processing unit 11 adds individual motion noise to the human body model based on the human motion noise model 21 to generate individual differences in motion. Specifically, for example, for each joint 71, 72, 73 of the joint model 70, time series data of the joint angle is acquired from the motion capture database 51. Individual differences in motion are expressed by expanding and contracting the time-series data of the joint angles. The range of the expansion / contraction rate of the time series data of the joint angle is set in a range of 1/2 to 2 times the original data. The probability that the expansion / contraction rate appears is assumed to be constant. The probability that the original data appears, the probability that the half data appears, and the probability that the double data appears are the same. Based on the above rules, individual differences in motion are expressed by adding noise to the time-series data of joint angles.

  In addition, when it is assumed that a distance image is generated by a stereo camera in the real space, the noise generation processing unit 11 applies a preset error model to the stereo matching position of the image, and generates a distance image. Reproduce the measurement error. That is, in the virtual experiment environment, an accurate correspondence between corresponding points of stereo images output from the set stereo camera can be determined based on camera specifications and setting of the viewpoint position setting processing unit 12 described later. An error is added within a circle having a radius of 3 pixels around the stereo corresponding point. At that time, assuming that the probability density function in which the stereo measurement points including the error are distributed around the correct stereo correspondence point follows the normal distribution, ± 3 pixels around the accurate stereo correspondence point is ± 3σ (standard deviation) = 1 pixel) to apply a noise model to reproduce the measurement error.

  In addition, in the real space, there may be a concealment between the subject and the camera. In this embodiment, the noise generation processing unit 11 causes the noise generation processing unit 11 to select an appropriate position between the subject (human body model) and the camera. In addition, the concealment phenomenon is reproduced by arranging one or more rectangular parallelepipeds of appropriate size. In the real space, there is a case where there is an obstacle between the light source and the subject and a shadow is reflected on the subject, or there are cases where light sources of various colors are mixed and appear on the subject. In the present embodiment, these phenomena are treated as illumination noise, and the noise generation processing unit 11 reproduces this phenomenon by projecting a preset illumination noise model 25 onto a subject in the virtual space. Specifically, illumination noise is generated by adding luminance noise generated by changing the amplitude of a sine wave of multiple frequencies within a range of ± 3.3% of the gradation of the luminance image to the luminance image. . In addition, not only luminance values but also illumination noises of various colors are generated in the same manner. As described above, the noise generation processing unit 11 adds the noise of the shielding object or the noise of the illumination to the human body model.

  Thereafter, the image processing apparatus 1 constructs the recognition database 54 by sequentially generating images corresponding to the captured images in the real space from the three-dimensional shape data (human body model) generated by adding noise as described above. . First, the image processing apparatus 1 uses the camera specification and viewpoint position setting processing unit 12 to set shooting conditions regarding the camera, viewpoint position, and the like of an image acquired from a human body model. Specifically, for example, sensor values (luminance image, distance image) output by the camera, its focal length, image surface size, image surface resolution, camera installation state (position, height, roll angle, pan angle, tilt) Angle, etc.) and the relative positional relationship with the subject. At that time, a plurality of cameras can be set. Subsequently, based on the three-dimensional shape data (human body model), the image generation processing unit 13 generates an image such as luminance and distance. At that time, based on the shooting conditions set by the camera specification and viewpoint position setting processing unit 12, the human body model is centrally projected on the image plane with the distance or luminance to be acquired in accordance with the setting of the camera specification and the viewpoint position. Thus, an image based on the shooting conditions is generated. Here, a pinhole camera model is adopted as the camera model, but it can also be applied to other camera models such as a fisheye camera model. As described above, the image generation processing unit 13 generates an image from the human body model and stores each image in a corresponding folder in the image memory 30.

  Further, the image processing apparatus 1 uses the image feature calculation processing unit 14 based on each image generated from the human body model to include in the image necessary for each human recognition process such as the range, position, posture, and motion of the person in the image. The image feature of the person is calculated. As this image feature, various image features preset in the image feature calculation function group 40 are calculated. Based on the calculated image features, the image processing apparatus 1 performs machine learning on the machine features by the machine learning unit 15 to construct a recognition database. This is stored as a recognition database 54 used for human recognition.

  At this time, for example, (1) image features for extracting a human region in the image are calculated, and humanity is machine-learned from the image features for extracting the human region. (2) Image features for estimating the posture of a person are calculated, and the rotation angle and position of each joint are machine-learned from the image features for estimating the posture. (3) An image feature for determining a human motion is calculated, and an image feature for determining a human motion is calculated. An example of image feature calculation and recognition database construction in (1), (2), and (3) will be described below. In this embodiment, an example of using a distance image is shown, but the present invention can also be applied to a luminance image and a binarized image.

(1) Constructing an extraction database for human areas.
For the extraction of the human region, the characteristics of the size, length, and circularity of the object acquired from the distance image are used as the characteristics of humanity. A recognition database is constructed by learning whether these image features are human or non-human by using a multilayer perceptron. The calculation of the image feature of size, length, and circularity is not particularly limited, and is calculated using existing image processing technology. For example, the size can be obtained by integrating the pixel values of the area occupied by the object on the distance image. The length can be easily obtained by dividing the previously obtained size amount by the number of pixels occupied by the object on the distance image. As for the circularity, it is possible to obtain the circularity amount by applying a circular Hough transform to the outermost edge line of the object. These image characteristics of humanity are examples, and can be applied to other than these. The various image feature calculation methods described above can also be applied using existing image processing techniques.

(2) About construction of human posture recognition database.
For the posture recognition of a person, the characteristics of the size and length of a target in a local area obtained by dividing a distance image into small grid areas are used. Using the size and length of the object in the local region on the distance image as input, a recognition database is constructed by learning each joint angle with a multi-layer perceptron with supervision. The size and length characteristics of the object in the local area on the distance image are calculated in the same manner as the method described in (1).
(3) Constructing a human motion recognition database.
The human motion is constructed by learning the time series data of each joint angle and displacement estimated using the human posture recognition database constructed in (2) using an HMM (Hidden Markov Model) algorithm.

  The image processing apparatus 1 performs machine learning on the image features calculated in this way, and constructs a recognition database 54 for recognizing a person in the image. Although an example of calculating the image feature from the distance image is shown here, application to a luminance image or a binarized image is also possible. Also, the image feature can be applied by adding a new image feature calculation function as needed, instead of applying only to a predetermined image feature.

Next, the procedure of image processing by the image processing apparatus 1 will be described.
FIG. 4 is a flowchart showing a procedure of image processing by the image processing apparatus 1.
As shown in the figure, the image processing apparatus 1 acquires human three-dimensional shape data and motion capture data from each of the databases 50 and 51 (S101), and first applies a joint model 70 to the three-dimensional shape data to form a human body model. Is created (S102). Next, based on information about the rotation angle and position of the human joint at the time of a specific action or posture read out from the motion capture database 51 (displacement information for each joint), the joint model 70 is changed to change each of the human body models. The joint is moved to change the human body model (S103). In addition, the noise generation processing unit 11 causes differences in human body models according to real space, such as human posture, motion, physique, clothing pattern, clothing shape and thickness, lighting, and shielding, as described above. In addition, individual differences and the like are generated (S104), and each three-dimensional shape data (human body model) is generated.

  Subsequently, shooting conditions relating to the camera or the like of the image desired to be acquired from the human body model are set (S105), and based on the setting, processing is performed to generate an image from the human body model to which noise or the like is added (S106). The image is stored in the image memory 30. Next, based on each generated image, an image feature is calculated by the image feature calculation processing unit 14 (S107), and machine learning is performed by the machine learning unit 15 (S108). Further, the image processing apparatus 1 repeats the above procedure (S101 to S108), generates an image by changing a person's movement, posture, noise, shooting conditions, etc., and sequentially machine-learns the image features, A recognition database and an image recognition system necessary for recognizing a person in an actual image are constructed.

  As described above, in this embodiment, an image (for example, an image that is difficult to acquire in the real space is projected by projecting data such as a few human shapes, postures, and movements acquired in the real space onto the three-dimensional virtual space. , Images taken from a plurality of viewpoints) can be easily generated and acquired. In addition, by applying a noise model that is statistically obtained through experiments to human shapes, postures, and movements in virtual space, differences in human shapes, postures, movements, etc. in real space are added. A large amount of data can be generated by acquiring the three-dimensional shape data close to reality. At that time, by reproducing a preset illumination model in the virtual space for shadows, disturbance light, and illumination, a large amount of 3D shape data in which a phenomenon closer to the phenomenon of light in the real space has occurred is generated. be able to.

  Further, as described above, the acquired enormous three-dimensional shape data can be used as a multi-view distance image, a single-view distance image, a multi-view luminance image, a single-view luminance image, a multi-view binary image, a single-view binary image, etc. An image that can be converted into a kind of image and that is comparable to an image acquired in the real space can be acquired. Further, by constructing a recognition database by machine learning of predetermined image features calculated from the images acquired as described above, a highly versatile recognition database can be constructed using only data acquired in the virtual space. By applying this recognition database to the real space, it is possible to construct an image recognition system that does not cause a difference in the types of photographing devices. That is, even if the photographing device is a distance image sensor or a luminance image sensor, it is possible to recognize a person or an action in the image regardless of the difference. For example, the image characteristics can be obtained from a two-dimensional luminance image of the luminance image sensor. By calculating and retrieving the corresponding distance image from the three-dimensional shape data, a recognition result in a higher-dimensional image can be acquired. By referring to the result, it is possible to recognize a human shape, posture, movement, etc. with higher accuracy.

  The present invention can also be realized as a program for realizing the above-described units of the image processing apparatus 1 by a computer.

  DESCRIPTION OF SYMBOLS 1 ... Image processing apparatus, 10 ... Image processing part, 11 ... Noise generation processing part, 12 ... Camera specification and viewpoint position setting processing part, 13 ... Image generation processing part, 14 ... Image feature calculation processing unit, 15 ... machine learning unit, 20 ... noise generation function group, 30 ... image memory, 40 ... image feature calculation function group, 50 ... 3D human shape Database 51... Motion capture database 54. Recognition database 70. Human joint model.

Claims (7)

Means for acquiring human three-dimensional shape data;
Means for applying a three-dimensional shape data to a human joint model in a virtual space to create a human body model in the virtual space;
Means for moving each joint of the human body model in accordance with the movement of each joint of the joint model and changing the human body model;
Means for adding noise corresponding to the difference in real space to the human body model;
Means for generating an image from a human body model;
Means for storing an image generated from a human body model;
An image processing apparatus comprising: The image processing apparatus according to claim 1,
Means for acquiring displacement information for each joint of the joint model according to the movement and posture of the person;
Means for changing the joint model based on displacement information for each joint;
An image processing apparatus comprising: In the image processing apparatus according to claim 1 or 2,
Means for setting image capturing conditions of an image acquired from a human body model,
An image processing apparatus, wherein means for generating an image from a human body model generates an image based on a photographing condition from the human body model. The image processing apparatus according to any one of claims 1 to 3,
An image processing apparatus characterized in that the means for adding noise includes means for adding at least one noise of a human posture, motion, and physique to a human body model to generate an individual difference. In the image processing device according to any one of claims 1 to 4,
An image processing apparatus characterized in that the means for adding noise has means for adding at least one noise of clothes, lighting, and shielding to a human body model. The image processing apparatus according to any one of claims 1 to 5,
Means for calculating an image feature of a person in the image based on an image generated from a human body model;
Means for constructing a recognition database for machine learning of image features to recognize people in the image;
An image processing apparatus comprising:   The program for implement | achieving each means of the image processing apparatus as described in any one of Claim 1 thru | or 6 with a computer.

Images