JP4407244B2 - Robot apparatus and object learning method thereof - Google Patents

Description

  The present invention relates to a robot apparatus that learns visual features of an object that exists in an environment through dialogue with a user and an object learning method thereof.

  Conventionally, an object recognition system mounted on a robot apparatus or the like has an object learning phase for learning visual characteristics of an object and an object recognition phase for recognizing a learned object based on the learning result.

  In this case, in the object learning phase, for example, under an ideal environment where the background and the lighting environment are known, a learning target object selected in advance by a human is captured by a camera or the like mounted on the object recognition system, and the camera Based on the image data output from the device, a method for causing the apparatus to learn the characteristics of the object is widely used (see, for example, Patent Document 1 below).

Japanese Patent Laid-Open No. 9-237340

  By the way, generally in such a conventional object learning phase, an object region is extracted as a pre-processing using an image region dividing method, and feature extraction is performed based on the result. However, there is a problem that satisfactory object learning cannot be expected when region changes in various ways.

  Further, in the conventional object learning phase, there is a problem that the user has to operate the robot apparatus by performing a predetermined procedure using a user interface such as a keyboard, a mouse, or a touch panel. On the other hand, in order for the robot apparatus to enter a general home environment, it is necessary that an operation procedure for the robot apparatus during object learning does not become a burden on the user. For example, it is preferable from the viewpoint of the user's burden if the robot apparatus can learn an object through dialogue.

  The present invention has been proposed in view of such conventional situations, and a robot apparatus for accurately learning an object in the environment through dialogue with a user even in an actual environment where the background changes in various ways, and its The object is to provide an object learning method.

In order to achieve the above-described object, the robot apparatus according to the present invention recognizes voice in a robot apparatus that learns the visual characteristics of an object existing in the environment, and receives an instruction to start object learning from the user. When it is determined that an instruction to start object learning is given from the user by the voice recognition means, the voice recognition means for determining whether or not it is given by voice , An audio output means for outputting a sound for instructing to present a learning target object to be learned while moving, an imaging means for imaging the learning target object, and an object region image occupied by the learning target object in the captured image Extracting image extracting means, and features for extracting and holding a feature quantity group consisting of a plurality of feature quantities from the object region image extracted by the image extracting means And a holding means, the feature quantity retention means, by evaluating the results of feature extraction from the object-area image, whether the learning object by the user is presented under appropriate circumstances If it is determined that it is presented under an inappropriate situation, the feature output group extracted from the object region image is not retained, and the audio output means Outputs a voice instructing the user to re-present under the appropriate circumstances.

  Here, the image extraction means extracts a motion region image that is a region that has changed before and after moving the learning target object from the captured image, and the user who has moved at least the learning target object from the motion region image The object region image is extracted by masking the region corresponding to the hand portion of the.

  When learning such a visual feature of an object existing in the external environment, such a robot apparatus first gives an instruction to the user by voice so that the learning target object to be learned is moved. Then, an image of the user moving the learning target object is picked up, an object region image of the learning target object is extracted from the picked-up image, and a feature amount group is extracted from the object region image and held.

  The robot apparatus further includes a feature amount extracting unit that extracts a feature amount group from an input image captured by the imaging unit, a feature amount group of a learned object held in the feature amount holding unit, and the feature The image processing apparatus may further include a feature amount comparison unit that compares the feature amount group extracted by the amount extraction unit and detects whether or not the learned object exists in the input image.

  In this case, the robot apparatus extracts a feature amount group from the input image captured by the imaging unit, and compares the feature amount group with a learned feature amount group held in the feature amount holding unit. Then, it is detected in real time whether or not a learned object exists in the input image.

In order to achieve the above-described object, the object learning method according to the present invention is an object learning method of a robot apparatus that learns visual features of an object existing in an environment, and recognizes speech and learns an object from a user. A voice recognition step for determining whether or not an instruction to start the object has been given by voice, and if the user gives an instruction to start object learning in the voice recognition step, the user A presentation instruction step for outputting a sound for instructing to move and present a learning target object as a learning target as an object learning procedure, an imaging step for imaging the learning target object by an imaging means, and in the captured image An image extraction step for extracting an object region image occupied by the learning target object, and a plurality of feature amounts from the object region image extracted in the image extraction step Have a feature amount holding step of holding by extracting the symptoms amount group, in the feature amount storing step, to assess the results of feature extraction from the object-area image and the learning target object by the user It is determined whether or not it is presented in an appropriate situation, and if it is judged that it is presented in an inappropriate situation, the voice output is performed without retaining the feature amount group extracted from the object region image. The means outputs a voice instructing the user to re-present the learning target object under an appropriate situation.

  Here, in the image extraction step, a motion region image that is a region that has changed before and after the learning target object is moved is extracted from the captured image, and at least the learning target object is moved from the motion region image. The object region image is extracted by masking the region corresponding to the hand portion of the.

  In such an object learning method, when the robot apparatus learns the visual characteristics of an object existing in the external environment in which the robot apparatus acts, the user first makes a voice call so that the learning target object to be learned is moved. Give instructions. The robot apparatus captures an image of the user moving the learning target object, extracts an object area image of the learning target object from the captured image, and extracts and holds a feature amount group from the object area image.

  Further, the object learning method includes a feature amount extraction step of extracting a feature amount group from the input image captured by the imaging unit, a feature amount group of the learned object held in the feature amount holding step, It may further include a feature amount comparison step of comparing the feature amount group extracted in the feature amount extraction step and detecting whether or not the learned object is present in the input image.

  In this case, in this object learning method, the feature amount group is extracted from the input image captured by the imaging unit of the robot apparatus, and the feature amount group is stored in the feature amount group of the learned object held in the feature amount holding step. By comparing, it is detected in real time whether there is a learned object in the input image.

  According to the robot apparatus and the object learning method of the present invention, when learning the visual characteristics of an object existing in the external environment in which the robot apparatus acts, first, the learning object to be learned is moved. The user is instructed by voice to capture the image of the user moving the learning target object, the object region image of the learning target object is extracted from the captured image, and the feature quantity group is extracted from the object region image In this way, objects in the environment can be accurately learned even in a real environment where the background changes in various ways. In particular, since the object can be learned through dialogue with the user, the burden on the user learning the object is reduced, and the entertainment characteristics of the robot apparatus are also improved.

  In addition, a feature amount group is extracted from the input image captured by the image pickup unit of the robot apparatus, and the feature amount group is compared with the feature amount group of the learned object stored in advance, so that it has been learned in the input image. Whether or not an object exists can be detected in real time.

  Hereinafter, specific embodiments to which the present invention is applied will be described in detail with reference to the drawings. In this embodiment, the present invention is applied to a biped walking type robot apparatus that learns an object existing in an external environment. This robot apparatus is a practical robot that supports human activities in various situations in the living environment and other daily lives, and is also an entertainment robot that can express basic actions performed by humans.

(1) Configuration of Robot Device As shown in FIG. 1, in the robot device 1 according to the present embodiment, the head unit 3 is connected to a predetermined position of the trunk unit 2, and the left and right arm unit 4R. / L and two right and left leg units 5R / L are connected to each other (provided that R and L are suffixes indicating right and left, respectively, and the same applies hereinafter).

  The joint degree-of-freedom configuration of the robot apparatus 1 is schematically shown in FIG. The neck joint that supports the head unit 3 has three degrees of freedom: a neck joint yaw axis 101, a neck joint pitch axis 102, and a neck joint roll axis 103.

  Each arm unit 4R / L constituting the upper limb includes a shoulder joint pitch axis 107, a shoulder joint roll axis 108, an upper arm yaw axis 109, an elbow joint pitch axis 110, a forearm yaw axis 111, and a wrist. A joint pitch axis 112, a wrist joint roll axis 113, and a hand part 114 are included. The hand portion 114 is actually a multi-joint / multi-degree-of-freedom structure including a plurality of fingers. However, since the operation of the hand 114 has little contribution or influence on the posture control or walking control of the robot apparatus 1, it is assumed in this specification that the degree of freedom is zero. Therefore, it is assumed that each arm portion has seven degrees of freedom.

  The trunk unit 2 has three degrees of freedom: a trunk pitch axis 104, a trunk roll axis 105, and a trunk yaw axis 106.

  Each leg unit 5R / L constituting the lower limb includes a hip joint yaw axis 115, a hip joint pitch axis 116, a hip joint roll axis 117, a knee joint pitch axis 118, an ankle joint pitch axis 119, and an ankle joint. A roll shaft 120 and a foot 121 are included. In the present specification, the intersection of the hip joint pitch axis 116 and the hip joint roll axis 117 defines the hip joint position of the robot apparatus 1. The foot 121 of the human body is actually a structure including a multi-joint / multi-degree-of-freedom sole, but the foot of the robot apparatus 1 has zero degrees of freedom. Accordingly, each leg is configured with 6 degrees of freedom.

  In summary, the robot apparatus 1 as a whole has a total of 3 + 7 × 2 + 3 + 6 × 2 = 32 degrees of freedom. However, the robot device 1 for entertainment is not necessarily limited to 32 degrees of freedom. Needless to say, the degree of freedom, that is, the number of joints, can be increased or decreased as appropriate in accordance with design / production constraints or required specifications.

  Each degree of freedom of the robot apparatus 1 as described above is actually implemented using an actuator. It is preferable that the actuator be small and light in light of demands such as eliminating the appearance of extra bulges on the appearance and approximating the shape of a human body, and performing posture control on an unstable structure such as biped walking. .

  FIG. 3 schematically shows a control system configuration of the robot apparatus 1. As shown in FIG. 3, the control system is a motion that controls the whole body cooperative motion of the robot apparatus 1 such as driving the actuator 350 and the actuator 350 and the thought control module 200 that dynamically controls emotion judgment and emotional expression in response to user input. And a control module 300.

  The thought control module 200 includes a CPU (Central Processing Unit) 211, a RAM (Random Access Memory) 212, a ROM (Read Only Memory) 213, and an external storage device (hard disk) that execute arithmetic processing related to emotion judgment and emotion expression. A drive or the like) 214 and is an independent drive type information processing apparatus capable of performing self-contained processing in a module.

  The thought control module 200 determines the current emotion and intention of the robot device 1 according to stimuli from the outside such as image data input from the image input device 251 and sound data input from the sound input device 252. Here, the image input device 251 includes, for example, one CCD (Charge Coupled Device) camera on each side, and the audio input device 252 includes, for example, a plurality of microphones. Moreover, the thought control module 200 can output a voice via a voice output device 253 provided with a speaker.

  In addition, the thought control module 200 issues a command to the motion control module 300 to execute an action or action sequence based on decision making, that is, exercise of the limbs.

  One motion control module 300 includes a CPU 311 for controlling the whole body cooperative motion of the robot apparatus 1, a RAM 312, a ROM 313, and an external storage device (hard disk drive, etc.) 314, and performs self-contained processing within the module. It is an independent drive type information processing apparatus that can be performed. The external storage device 314 can store, for example, walking patterns calculated offline, target ZMP trajectories, and other action plans. Here, the ZMP is a point on the floor where the moment due to the floor reaction force during walking is zero, and the ZMP trajectory is, for example, a trajectory in which the ZMP moves during the walking operation period of the robot apparatus 1. Means. Regarding the concept of ZMP and the point where ZMP is applied to the stability criteria for walking robots, “LEGGED LOCOMOTION ROBOTS” written by Miomir Vukobratovic (“Walking Robot and Artificial Feet” written by Ichiro Kato (Nikkan Kogyo Shimbun)) It is described in.

  The motion control module 300 includes an actuator 350 that realizes the degrees of freedom of joints distributed throughout the body of the robot apparatus 1 shown in FIG. 2, a posture sensor 351 that measures the posture and inclination of the trunk unit 2, and left and right soles Various devices such as ground check sensors 352 and 353 for detecting the leaving or landing of the vehicle and a power supply control device 354 for managing the power supply of the battery or the like are connected via a bus interface (I / F) 301. Here, the posture sensor 351 is configured by, for example, a combination of an acceleration sensor and a gyro sensor, and the grounding confirmation sensors 352 and 353 are configured by proximity sensors, micro switches, or the like.

  The thought control module 200 and the motion control module 300 are constructed on a common platform, and are interconnected via bus interfaces 201 and 301.

  The motion control module 300 controls the whole body cooperative motion by each actuator 350 in order to embody the action instructed from the thought control module 200. That is, the CPU 311 extracts an operation pattern corresponding to the action instructed from the thought control module 200 from the external storage device 314 or internally generates an operation pattern. Then, the CPU 311 sets a foot movement, a ZMP trajectory, a trunk movement, an upper limb movement, a waist horizontal position, a height, and the like according to the specified movement pattern, and commands that instruct the movement according to these setting contents. The value is transferred to each actuator 350.

  In addition, the CPU 311 detects the posture and inclination of the trunk unit 2 of the robot apparatus 1 from the output signal of the posture sensor 351, and each leg unit 5R / L is caused to move freely by the output signals of the ground contact confirmation sensors 352 and 353. Alternatively, the whole body cooperative movement of the robot apparatus 1 can be adaptively controlled by detecting whether the robot is standing or standing.

  Further, the CPU 311 controls the posture and operation of the robot apparatus 1 so that the ZMP position is always directed to the center of the ZMP stable region.

  Furthermore, the motion control module 300 returns to the thought control module 200 the level of behavior as intended as determined by the thought control module 200, that is, the processing status.

  In this way, the robot apparatus 1 can determine its own and surrounding conditions based on the control program and act autonomously.

(1-2) Voice Dialog Function of Robot Device Next, the voice dialog function of the robot device 1 described above will be described with reference to the functional block diagram of FIG.

  As shown in FIG. 4, the part related to the voice interaction function of the robot apparatus 1 generates a plurality of types of responses based on the voice recognition unit 400 that recognizes the user's speech and the recognition result of the voice recognition unit 400. A response generation unit 410; a dialogue control unit 420 that selects one response from a plurality of types of responses generated by the response generation unit 410; and a voice synthesis unit that generates a voice signal of the response selected by the dialogue control unit 420. 430.

  The voice recognition unit 400 recognizes the user's utterance content based on the voice data D1 input from the voice input device 252 (FIG. 3), and uses the recognition result as character string data (hereinafter, “recognized character string data”). It is supplied to the response control unit 410 via the dialogue control unit 420 as D2. The recognized character string data D2 includes para-original words such as prosody information such as intonation and accents obtained at the time of recognition processing, and is, for example, a prosody for asking a question about the prosody of the utterance of the user utterance. Is determined, the character “?” Is added to the end of a series of character strings for the utterance.

  The response generation unit 410 includes response generation modules 410a, 410b,... That generate various responses to user utterances. Each of the response generation modules 410a, 410b,... Has data of rules (hereinafter referred to as “response generation rules”) that predetermine what responses should be returned according to the content of the user's utterance. When the recognized character string data D2 is supplied from the voice recognition unit 400, a response to the user's utterance content obtained based on the recognized character string data D2 is generated according to the corresponding response generation rule, and the character string data ( Hereinafter, it is referred to as “response character string data.”) D3a, D3b... Are supplied to the dialogue control unit 420.

  When the response character string data D3a, D3b,... Are supplied from the response generation modules 410a, 410b,... Of the response generation unit 410, the dialogue control unit 420 selects one of the response character string data D3a, D3b,. This is supplied to the synthesis unit 430.

  The voice synthesis unit 430 generates a voice signal D5 by performing voice synthesis processing based on the response data D4 supplied from the dialogue control unit 420, and outputs this to the voice output device (FIG. 3). As a result, a response corresponding to the response data D4 is output from the robot apparatus 1 to the outside based on the audio signal D5.

(1-3) Autonomous Behavior Function of Robot Device Subsequently, the autonomous behavior function of the robot device 1 described above will be described with reference to the schematic diagram of FIG. In FIG. 5, circles represent entities called objects or processes. As a result of these objects communicating asynchronously, the entire system operates. Each object performs data transfer and invoke by message communication and inter-object communication using a shared memory.

  As shown in FIG. 5, the part relating to the autonomous behavior function of the robot apparatus 1 includes a kinematics odometry (Kinematics Odometry) KINE 500, a plane extractor PLEX 510, a grid occupancy calculation unit (Occupancy Grid) OG 520, A landmark sensor unit (Landmark Sensor) CLS 530, an absolute coordinate calculation unit (Localization) LZ 540, and a route plan determination unit (Situated Behavior Layer) SBL 550 are processed in units of objects. The operation of each object will be described in detail with reference to the flowchart of FIG.

  The kinematic odometry KINE 500 shown in FIG. 5 receives image data D11 captured by the left and right CCD cameras and sensor data D12 such as a joint angle of the robot apparatus 1. In step S1, the kinematic odometry KINE 500 receives the image data D11 and the sensor data D12, and updates the previous image data and sensor data stored in the memory.

  Next, in step S2, kinematic odometry KINE 500 takes time correspondence between image data D11 and sensor data D12. Specifically, the joint angle of the sensor data D12 at the time when the image of the image data D11 is captured is determined.

  Subsequently, at step S3, the kinematic odometry KINE 500 uses the joint angle data to set the robot center coordinate system centered on the robot apparatus 1 to the camera coordinate system centered on the CCD camera provided in the head unit 3. Convert to At this time, the kinematic odometry KINE 500 derives a homogeneous transformation matrix or the like of the camera coordinate system from the robot center coordinate system, and supplies the homogeneous transformation matrix D13 and image data corresponding thereto to an object that performs image recognition. . Specifically, parallax image data D14 obtained from the homogeneous transformation matrix D13 and the left and right image data is supplied to the plane extraction unit PLEX 510, and the homogeneous transformation matrix D13 and the image data D15 are supplied to the landmark sensor unit CLS 530. . The kinematic odometry KINE 500 obtains the movement amount of the robot apparatus 1 from the walking parameter obtained from the sensor data D12 and the count value of the number of steps using the ground contact confirmation sensors 352 and 353 (FIG. 3), and the robot of the robot apparatus 1 A movement amount in the central coordinate system (hereinafter also referred to as “odometry”) is calculated. The kinematic odometry KINE 500 supplies the odometry D16 to the grid occupancy rate calculation unit OG520 and the absolute coordinate calculation unit LZ540.

  Subsequently, in steps S4 and S5, the plane extraction unit PLEX 510 and the grid occupancy rate calculation unit OG 520 calculate and update the grid occupancy rate (environment map) of the obstacle, and the landmark sensor unit CLS 510 and the absolute coordinate calculation unit LZ 540 Calculate and update absolute coordinates.

  Here, the processing in the plane extraction unit PLEX 510 and the grid occupancy rate calculation unit OG 520 and the processing in the landmark sensor unit CLS 530 and the absolute coordinate calculation unit LZ 540 in steps S4 and S5 are respectively shown in FIGS. ) As shown in FIG.

  That is, when the obstacle grid is calculated / updated, the plane extraction unit PLEX 510 firstly converts the homogeneous transformation matrix D13 which is kinematic data calculated by the kinematic odometry KINE 500 in step S11 of FIG. When the parallax image data D14 corresponding to this is input, the data stored in the memory until then is updated.

  Next, in step S12, the plane extraction unit PLEX 510 calculates three-dimensional position data (range data) from the parallax image data D14 using a calibration parameter or the like of the CCD camera (for example, see Japanese Patent Application Laid-Open No. 2000-350239). reference). The plane extraction unit PLEX 510 extracts a plane excluding a plane such as a wall or a table as a plane by using Hough transform or the like from the range data. In step S13, the plane extraction unit PLEX 510 selects a floor surface from the homogeneous transformation matrix D13 so as to correspond to the plane on which the sole of the robot apparatus 1 is grounded, and is not on the floor surface, for example, a predetermined point An object at a position higher than the threshold value is detected as an obstacle, and this obstacle information D17 is supplied to the grid occupancy rate calculation unit 520.

  Subsequently, in step S14, the grid occupancy rate calculation unit OG520 receives the odometry calculated by the kinematic odometry KINE500 and the observation information (obstacle information) of the obstacle calculated by the plane extraction unit PLEX510. And the previous data stored in the memory is updated. In step S15, the grid occupancy rate calculation unit OG520 updates the grid occupancy rate of the obstacle that holds the probability of whether there is an obstacle on the floor surface by a probabilistic method.

  The grid occupancy rate calculation unit OG520 holds, for example, an environment map indicating obstacle information (grid occupancy rate of obstacles) around 4m around the robot device 1 and posture information indicating the direction in which the robot device 1 faces. is doing. Here, as shown in FIG. 8, for example, the environment map is obtained by dividing the surrounding environment around the robot apparatus 1 into a plurality of grids, and each grid is a grid in which an obstacle exists (in the figure). The grid is classified into a black grid, an obstacle-free grid (a white grid in the figure), and an unobserved grid (a hatched grid in the figure). The grid occupancy rate calculation unit OG520 updates such an environment map and supplies the updated recognition result (obstacle information D19) to the route plan determination unit SBL550, whereby the upper layer, that is, the route plan determination unit SBL550. Plans to avoid obstacles can be created (for example, “Sabe, Ohashi, Kawamoto, Gutman, Fukuchi, Kasumigahara,“ Obstacle Avoidance and Walking Plan Using Stereo Images by Biped Robot ”, (Refer to Proceedings of the 8th Image Sensing Symposium p.237-242).

  On the other hand, when calculating / updating the absolute coordinates, the landmark sensor unit CLS 530 first receives the kinematics data Homogeneous transformation matrix D13 and image data D15 from the kinematic odometry KINE 500 in step S21. These data stored in the memory are updated. Then, image processing of the image data D15 is performed to detect a color landmark recognized in advance. The position and size of the color landmark on the image data D15 are converted into a position in the camera coordinate system. Further, in step S22, the landmark sensor unit CLS 530 uses the homogeneous transformation matrix D13 to convert the position of the color landmark in the camera coordinate system to the position in the robot center coordinate system, and the color landmark in the robot center coordinate system. Position information (color landmark relative position information) D18 is supplied to the absolute coordinate calculation unit LZ540.

  That is, the landmark sensor unit CLS 530 uses, for example, sensor information of the robot apparatus 1 and operation information performed by the robot apparatus 1 in an environment including artificial color landmarks having colors such as green, pink, and blue. The self-position (position and posture) of the robot apparatus 1 is identified.

  Subsequently, in step S23, when the odometry D16 from the kinematic odometry KINE500 and the color landmark relative position information D18 from the landmark sensor unit CLS530 are input, the absolute coordinate calculation unit LZ540 is stored in the memory in advance. Update these data.

  Subsequently, in step S24, the absolute coordinate calculation unit LZ540 uses the absolute coordinates (positions in the camera coordinate system) of the color landmarks recognized in advance, the color landmark relative position information D18, and the odometry D16, and uses the probabilistic method. Thus, the absolute coordinate (position in the camera coordinate system) of the robot apparatus 1 is calculated. Then, this absolute coordinate position D20 is supplied to the route plan determination unit SBL550.

  Returning to FIG. 6, when the obstacle information D19 is input from the grid occupancy rate calculation unit OG520 and the absolute coordinate position D20 is input from the absolute coordinate calculation unit LZ540, the route plan determination unit SBL550 is stored in the memory in advance. Update these data. In step S6, the route plan determination unit SBL550 acquires the recognition result regarding the obstacle existing around the robot apparatus 1 based on the obstacle information D19 from the grid occupancy rate calculation unit OG520, and from the absolute coordinate calculation unit LZ540. By acquiring the absolute coordinates of the current robot apparatus 1, a path that can be walked without colliding with an obstacle is generated with respect to a target point given in the camera coordinate system or the robot center coordinate system, Issue an operation command to perform the path. That is, the action to be taken by the robot apparatus 1 is determined from the input data according to the situation, and the action sequence is output.

(2) Object recognition system Next, the object recognition system mounted in the robot apparatus 1 mentioned above is demonstrated. The operation of this object recognition system includes an object learning phase for learning visual features of an unknown object and an object recognition phase for recognizing a learned object based on the learning result.

  Here, when learning and recognizing an object in an actual environment in which the robot apparatus 1 acts, an object recognition method that is robust against changes in the viewpoint and the lighting environment is required. Therefore, in the present embodiment, as an example of such an object recognition method, the method described in the specification and drawings of Japanese Patent Application No. 2003-124225 previously proposed by the present applicant is used, and an attempt is made to learn from now on. Feature points and their feature quantities are extracted from an image of an object (hereinafter referred to as “model image” as appropriate) and an input image (hereinafter referred to as “object image” as appropriate), and feature points having similar feature quantities are extracted. A pair (candidate corresponding feature point pair) is generated, and a model in the object image, that is, a learned object in the input image is recognized using the generated candidate corresponding feature point pair group.

  In practice, in the object learning phase, feature quantities at each feature point of a plurality of model images are learned in advance, and in the object recognition phase, feature quantities at the feature points of the input object image and a plurality of model images In the following, the feature amount extraction from the model image in the object learning phase and the feature amount extraction from the object image in the object recognition phase are performed in parallel below. Will be described.

  FIG. 9 shows a schematic configuration of the object recognition system in the present embodiment. In the object recognition system 600 shown in FIG. 9, the feature point extraction units 610a and 610b extract model feature points and object feature points from the model image and the object image, respectively, and the feature amount holding units 620a and 620b A feature amount, which will be described later, is extracted for the feature point and held together with the position information of the feature point. The feature quantity comparison unit 630 compares the feature quantity of each model feature point with the feature quantity of each object feature point to calculate the similarity or dissimilarity, and the feature quantities are similar using this similarity measure, A pair of model feature points and object feature points (candidate correspondence feature point pairs) that are highly likely to correspond is generated.

  The model posture estimation unit 640 detects the presence / absence of a model on the object image using the generated candidate corresponding feature point pair group, and in the case of the determination result of “model present”, “the detected model is the object image Under the constraint that “the image is deformed by affine transformation upward”, the operation of projecting the affine transformation parameters determined by three pairs randomly selected from the candidate corresponding feature point pair group to the parameter space is repeated, Among the clusters formed on the parameter space, each member of the cluster having the largest number of members is set as a true feature point pair (inlier), and an affine transformation parameter is obtained by least square estimation using this inlier. Then, the model posture estimation unit 640 outputs the model posture determined by this affine transformation parameter as a model recognition result.

  Hereinafter, each block of the object recognition system 600 will be described in detail. In the following description, the horizontal direction of the image is described as the X axis, and the vertical direction is described as the Y axis.

  The feature point extraction units 610a and 610b first perform smoothing filtering on the feature point extraction target image (model image or object image), for example, a convolution product (Gauss filtering) using a two-dimensional Gaussian function expressed by the following equation (1). A multi-resolution pyramid structure of an image is constructed by repeatedly and alternately applying image reduction by biquadratic linear interpolation resampling. Here, σ used in the Gaussian filter of Expression (1) is used as the resampling factor.

That is, as shown in FIG. 10, for example, a first level (highest resolution) image I ₁ is generated by applying a Gaussian filter g (x, y) with σ = √2 to the input image I, and further a Gaussian filter. To generate an image g * I ₁ . Then, the second level images I ₂ and g * I ₂ are generated by resampling the image g * I ₁ and applying a Gaussian filter. Similarly, the third level image I is generated from the image g * I ₂ . ₃ , g * I ₃ is generated.

Subsequently, the feature point extraction units 610a and 610b apply a DoG (Difference of Gaussian) filter to each level (each resolution) image. This DoG filter is a kind of second-order differential filter used for image edge enhancement, and approximates the processing that is performed until information from the retina is relayed by the outer knee in the human visual system. As a model, it is often used together with a LoG (Laplacian of Gaussian) filter. The output of the DoG filter can be easily obtained by taking the difference between the two Gaussian filter output images. That is, as shown in FIG. 10, an image DI ₁ (= I ₁ −g * I ₁ ) is obtained for the first level image, and an image DI ₂ (= I ₂ − for the second level and third level images is obtained. g * I ₂ ) and image DI ₃ (= I ₃ −g * I ₃ ) are obtained.

Then, the feature point extraction units 610a and 610b have a resolution within a predetermined range among local points (local maximum points and local minimum points) of the DoG filter output images DI ₁ , DI ₂ , DI _3. A point whose position does not change due to the change is detected as a feature point. Thereby, it is possible to realize matching between feature points that is robust to an enlargement / reduction operation of an image.

  Here, processing for detecting a feature point whose position does not change due to a change in resolution up to the Lth level of the multi-resolution pyramid structure, that is, a factor up to the (L−1) th power factor of σ will be described with reference to the flowchart of FIG. explain.

First, in step S31, local points (local maximum points and local minimum points) of the DoG filter output image DI ₁ at the first level (maximum resolution) are detected. For example, a 3 × 3 direct neighborhood can be used as the local neighborhood.

  Next, in step S32, for each detected local point, a corresponding point at the one higher level (one lower resolution layer) is obtained in consideration of image reduction accompanying resolution reduction. Determine whether or not. If it is a local point (Yes), the process proceeds to step S33, and if it is not a local point (No), the search is terminated.

  Subsequently, in step S33, it is determined whether or not the search has been successful up to the Lth level. If the search has not been performed up to the L-th level (No), the process returns to step S32 to perform a higher-level search. If the search has been successful up to the L-th level (Yes), the feature point is determined to be a feature point. The position information is held at.

For example, when detecting a feature point whose position does not change due to a resolution change up to the third level, as shown in FIG. 12, among the local points FP ₁ and FP ₂ detected in the first level image DI ₁ , FP ₁ is a feature point because corresponding points exist up to the image DI ₃ of the third level, and FP ₂ is not a feature point because corresponding points exist only up to the second level.

  Note that the feature point extraction units 610a and 610b may use a LoG filter instead of the DoG filter. Also, the corner- used in the corner detection of objects in the document “Harris C. and Stephens M.,“ A combined corner and edge detector. ”, In Proc. Alvey Vision Conf., Pp.147-151, 1988”. The output value of the ness function may be used instead of the DoG filter output.

Next, the feature quantity holding units 620a and 620b (FIG. 9) extract and hold the feature quantity for each feature point extracted by the feature point extraction units 610a and 610b. As the feature amount, density gradient information (gradient strength and gradient) of each point in the vicinity of the feature point derived from the image information of each level image (I _l , l = 1,..., L) of the multi-resolution pyramid structure. Gradient direction). The gradient strength M _{x, y} and gradient direction R _{x, y at the} point (x, y) are given by the following equations (2) and (3).

  Here, as the feature point vicinity region for calculating the feature amount, it is preferable to select a region that is symmetric with respect to the feature point so that the structure does not change with respect to the rotation change. Thereby, the robustness with respect to a rotation change is realizable. For example, (i) a method in which a region within a radius r pixel from a feature point is used as a feature point neighboring region, or (ii) a method in which a two-dimensional Gaussian weight symmetric about a feature point having a width σ around the feature point is applied to the density gradient. Can be used.

  FIG. 13A shows an example of density gradient information of the feature point neighboring area when the area within the radius of 3.5 pixels from the feature point is set as the neighboring area. Here, in FIG. 13A, the length of the arrow represents the gradient strength, and the direction of the arrow represents the gradient direction.

  The feature amount holding units 620a and 620b also hold a histogram (direction histogram) regarding the gradient direction in the vicinity of the feature point as the feature amount. FIG. 13B shows an example of the gradient direction histogram obtained from the density gradient information of FIG. Here, the class width Δθ in FIG. 13B is 10 deg, and the class number N is 36 (= 360 deg / 10 deg).

  Subsequently, the feature quantity comparison unit 630 (FIG. 9) compares the feature quantity of each model feature point with the feature quantity of each object feature point, and a pair of model feature points and object feature points having similar feature quantities ( A candidate corresponding feature point pair) is generated.

  Details of the processing in the feature amount comparison unit 630 will be described with reference to the flowchart of FIG. First, in step S41, the feature amount comparison unit 630 compares the direction histogram of each model feature point with the direction histogram of each object feature point to calculate the distance (difference) between the histograms and between the model and the object. The estimated rotation angle is obtained.

Here, two direction histograms H ₁ = {h ₁ (n), n = 1,..., N} and H ₂ = {h ₂ (n), where the class width Δθ and the class number N are the same. Assuming n = 1,..., N}, and h ₁ (n), h ₂ (n) indicate the frequencies in the class n, the distance d (between the histogram H ₁ and the histogram H ₂ H ₁ , H ₂ ) is given by, for example, the following formula (4). Here, r = 1, 2, ∞ is generally used as r in Equation (4).

  Using this equation (4), the degree of difference between the direction histograms of each model feature point and each object feature point is calculated. (I) Since the scale ratio between the model and the object is unknown at the matching stage, the model feature It is necessary to perform matching between direction histograms between each level of points and each level of object feature points. Further, regarding the matching between the direction histograms (ii), it is necessary to consider the rotational transformation amount between the model and the object.

Direction histogram H _m ^LV = {h _m ^LV (n), n = 1,..., N} at the level LV of the model feature point m, and direction histogram H _o ^lv = {h at the level lv of the object feature point o Consider a case where the degree of difference from _o ^lv (n), n = 1 ^,. Since the direction histogram changes cyclically with respect to the rotation transformation, the calculation of Equation (4) is performed while cyclically shifting the class of H _o ^lv one by one, and the minimum values are calculated as H _m ^LV and H _o ^lv. The degree of difference between At this time, the rotation angle of the object feature point can be estimated from the shift amount (shifted class number) when the minimum value of the dissimilarity is given. This method is known as a direction histogram intersection method.

^Assuming that a direction histogram obtained by shifting H _o ^lv by k classes is H _o ^{lv (k)} , the dissimilarity (H _m ^LV , H _o ^{lv (k)} ) between the direction histograms by the direction histogram crossing method is expressed by the following equation: It is given by (5).

Further, ^assuming that k ′ that gives the minimum d (H _m ^LV , H _o ^{lv (k)} ) is k ′, the estimated rotation angle θ (m, LV, o, lv) in the region near the object feature point o is It is given by equation (6).

Considering the above (i), the dissimilarity (H _m , H _o ) between the direction histograms of the model feature point m and the object feature point o is expressed by the following equation (7).

The feature quantity comparison unit 630 provides levels LV, lv () that give the minimum dissimilarity (H _m , H _o ) between the model feature point m and the object feature point o between the directional histograms (m m, n _o ). Hereinafter, the LV _m ^* and lv _o ^* ) and the estimated rotation angle θ (m, LV _m ^* , o, lv _o ^* ) are expressed as dissimilarity (H _m , H _o ) between the direction histograms. ).

Next, in step S42 (FIG. 14), the feature quantity comparison unit 630 selects K object feature points o _m1 ,..., O _mK in ascending order of the degree of difference between the direction histograms for each model feature point m. The candidate corresponding feature point pair is assembled. That is, K candidate corresponding feature point pairs (m, o _m1 ), ..., (m, o _mk ), ..., (m, o _mK ) are set for each model feature point m. Each candidate corresponding feature point pair (m, o _mk ) holds information of the corresponding level LV _m ^* , lv _omk ^* and the estimated rotation angle θ (m, LV _m ^* , o, lv _omk ^* ). The

  As described above, a pair group obtained by assembling candidate corresponding feature point pairs for all model feature points becomes a candidate corresponding feature point pair group.

  As described above, the feature amount comparison unit 630 does not accumulate the gradient strength in the histogram frequency, and simply focuses only on the gradient direction, and thus feature amount matching that is robust against changes in brightness is possible. Further, more stable matching can be performed in consideration of the shape of the direction histogram, and a secondarily stable feature amount (estimated rotation angle) can be obtained.

  In step S42 described above, it has been described that K candidate corresponding feature point pairs are selected for each model feature point m. However, the present invention is not limited to this. You may make it select all the pairs below.

  Here, the candidate corresponding feature point pair group generated by the above-described operation also includes corresponding point pairs that have similar direction histograms but have different spatial characteristics of the density gradient. Therefore, in the subsequent step S43 (FIG. 14), a pair is selected based on the similarity between density gradient vectors, and the candidate corresponding feature point pair group is updated.

Specifically, the density gradient vector at the level LV _m ^* in the vicinity of the model feature point m is U _m, and the density gradient at the level lv _omk ^* in the vicinity of the object feature point o that forms a corresponding point pair with the model feature point m. When the vector is U _o , pairs whose similarity between U _m and U _o falls below the threshold are excluded, and the candidate corresponding feature point pair group is updated.

FIG. 15 shows a method for calculating the similarity between the density gradient vectors U _m and U _o . First, U _m is spatially divided into four regions R _i (i = 1,..., 4), and an average density gradient vector v _i (i = 1,..., 4) of each region is obtained. The _{v i} in 8-dimensional vector V summarizes the _{U m} is expressed. Meanwhile, in order to perform the matching considering the density gradient information rotational transformation, _{U o} of the estimated rotation angle of the gradient direction previously obtained ^{_{θ (m, LV m *,}} o, lv omk *) is corrected by, _{U o} ^{Get *} . At this time, the value of the intermediate position is obtained by biquadratic linear interpolation. Similarly to the above, U _o ^* is divided into four regions R _i (i = 1,..., 4), and the average density gradient vector w _i (i = 1,..., 4) of each region is obtained. U _o is expressed by an eight-dimensional vector W in which w _i are combined. At this time, the similarity similarity (U _m , U _o ) ∈ [0, 1] between U _m and U _o is interpreted as the similarity between the average density gradient vectors V and W, for example, using a cosine correlation value. It calculates | requires by the following formula | equation (8). Here, in Expression (8), (V · W) represents the inner product of V and W.

  The feature amount comparison unit 630 obtains the similarity between the average density gradient vectors obtained by the above equation (8) for each candidate corresponding feature point pair, and selects a pair whose similarity is lower than the threshold δ as a candidate corresponding feature point pair group. The candidate corresponding feature point pair group is updated.

  As described above, the feature amount comparison unit 630 compares the feature amounts using the average density gradient vectors of the partial regions, and therefore, a density gradient caused by a slight shift in the feature point position or the estimated rotation angle or a change in brightness. Robust matching against information changes can be realized, and the amount of calculation can be reduced.

  Through the above operation, it is possible to extract a group of (model feature point-object feature point) pairs having similar local density gradient information in the vicinity of the feature point, but when viewed macroscopically, the pair thus obtained is extracted. The group includes “false feature point pairs (outliers)” in which the spatial positional relationship between corresponding feature points is inconsistent with the posture (model posture) of the model on the object image.

  Here, when there are three or more candidate corresponding feature point pairs, it is possible to estimate approximate affine transformation parameters by least square estimation, eliminate corresponding pairs in which the estimated model pose and spatial positional relationship contradict, and remain The model posture can be recognized by repeating the operation of estimating the model posture again with the pair.

  However, if there are many outliers in the candidate-corresponding feature point pair group or there are outliers that deviate significantly from the true affine transformation parameters, the estimation result by least squares estimation is generally not satisfactory. (Hartley R., Zisserman A., “Multiple View Geometry in Computer Vision.”, Chapter 3, pp.69-116, Cambridge University Press, 2000). In 640 (FIG. 9), under the affine transformation constraint, “true feature point pair (inlier)” is extracted from the spatial positional relationship of the candidate corresponding feature point pair group, and the model posture (parallel) is extracted using the extracted inlier. Estimate the movement amount, rotation, scaling, and affine transformation parameters that determine stretch.

  Hereinafter, processing in the model posture estimation unit 640 will be described. As described above, since the affine transformation parameters cannot be determined unless there are three or more candidate corresponding feature point pairs, when there are two or less candidate corresponding feature point pairs, the model posture estimation unit 640 has a model in the object image. No, or the model posture detection fails, “recognition is impossible” is output and the process is terminated. On the other hand, when there are three or more candidate corresponding feature point pairs, the model posture estimation unit 640 assumes that the model posture can be detected, and estimates the affine transformation parameters. Note that the model posture estimation unit 640 estimates the model posture based on the spatial positions of feature points at, for example, the first level (highest resolution) of the model image and the object image.

Here, the affine transformation of the model feature point [x y] ^{T to} the object feature point [u v] ^T is given by the following equation (9).

In this equation (9), a _i (i = 1,..., 4) represents parameters for determining rotation, enlargement / reduction, and stretch, and [b ₁ b ₂ ] ^T represents a translation parameter. Since there are six affine transformation parameters a ₁ ,..., A ₄ and b ₁ , b ₂ to be determined, the affine transformation parameters can be determined if there are three candidate corresponding feature point pairs.

A pair group P constituted by three pairs of candidate corresponding feature points is represented by ([x ₁ y ₁ ] ^T , [u ₁ v ₁ ] ^T ), ([x ₂ y ₂ ] ^T , [u ₂ v ₂ ] ^T ), ([X ₃ y ₃ ] ^T , [u ₃ v ₃ ] ^T ), the relationship between the pair group P and the affine transformation parameters can be expressed by the linear system shown in the following equation (10). .

  When this equation (10) is rewritten as Rx = s, the least squares solution of the affine transformation parameter x is given by the following equation (11).

  When the pair group P is repeatedly selected at random so that one or more outliers are mixed from the candidate corresponding feature point pair group, the affine transformation parameters are scattered and projected on the parameter space. On the other hand, when the pair group P composed only of inliers is selected repeatedly at random, the affine transformation parameters are all very similar to the true affine transformation parameters of the model pose, that is, close in the parameter space. Become. Therefore, when the pair group P is randomly selected from the candidate corresponding feature point pair group and the operation of projecting the affine transformation parameters onto the parameter space is repeated, the inlier has a high density (a large number of members) on the parameter space. ) Clusters will form and outliers will appear scattered. For this reason, clustering is performed on the parameter space, and the element of the cluster having the largest number of members becomes an inlier.

Details of processing in the model posture estimation unit 640 will be described with reference to a flowchart of FIG. As a clustering method in the model posture estimation unit 640, an NN (Nearest Neighbor) method is used. At this time, since b ₁ and b ₂ described above can take various values depending on the recognition target image, the selection of the clustering threshold in clustering depends on the recognition target even in the x space. Therefore, the model pose estimation unit 640 says that “there is almost no pair group P that gives affine transformation parameters that are similar to the true parameters and a ₁ ,..., A ₄ but different b ₁ , b ₂ ”. Under the assumption, clustering is performed only on the parameter space spanned by the parameters a ₁ ,..., A ₄ (hereinafter referred to as A). Even if a situation in which the above assumption does not hold, a problem can be easily avoided by performing clustering in the parameter space spanned by b ₁ and b ₂ independently of the A space and considering the result. it can.

First, initialization is performed in step S51 of FIG. Specifically, the count value cnt of the number of repetitions and cnt = 1, then select the pair group P ₁ at random from the candidate corresponding feature point pair group to obtain the affine transformation parameters A _1. Further, the cluster number N is set to N = 1, and a cluster Z ₁ centering on A ₁ is created on the affine transformation parameter space A. The centroid _{C 1} of this cluster _{Z 1} and _C 1 = _{A 1,} the number of members nz ₁ and _nz 1 = 1. The counter is set to cnt = 2, and the process proceeds to step S52.

Next, in step S52, the model posture estimation unit 640 randomly selects a pair group P _cnt from the candidate corresponding feature point pair group to obtain an affine transformation parameter A _cnt .

Subsequently, in step S53, the model posture estimation unit 640 performs clustering of the affine transformation parameter space using the NN method. Specifically, first, according to the following equation (12), the distance d (A _cnt , C _i ) between the affine transformation parameter A _cnt and the centroid C _i (i = 1,..., N) of each cluster Z _i. Among these, the minimum distance d _min is obtained.

Then, the cluster _{Z i} giving _{d min} if _{d min} <tau to a predetermined threshold value tau (e.g. tau = 0.1) to belong to _{A cnt,} the cluster _{Z i} with all members including the _{A cnt} to update the centroid _{C i.} Further, the number of members nz _i cluster _{Z i} and _nz i = _nz i +1. On the other hand, if d _min ≧ τ, a new cluster Z _{N + 1} having A _cnt as the centroid C _{N + 1} is created on the affine transformation parameter space A, the number of members nz _{N + 1 is set} to nz _{N + 1} = 1, and the number of clusters N is set to N = N + 1. And

  Subsequently, in step S54, it is determined whether or not a repeated end condition is satisfied. Here, as the repetition end condition, for example, when the maximum number of members exceeds a predetermined threshold (for example, 15) and the difference between the maximum number of members and the second largest number of members exceeds a predetermined threshold (for example, 3), Alternatively, it can be set to end when the count value cnt of the repetition number counter exceeds a predetermined threshold (for example, 5000 times). In step S54, if the repetition end condition is not satisfied (No), the count value cnt of the number of repetitions is set to cnt = cnt + 1 in step S55, and the process returns to step S52. On the other hand, if the repeated end condition is satisfied (Yes), the process proceeds to step S56.

  Finally, in step S56, the model posture estimation unit 640 estimates an affine transformation parameter for determining the model posture by the least square method using the inlier obtained above.

Here, the inliers ^{_{([x IN1 y IN1] T}} , [u IN1 v IN1] T), ([x IN2 y IN2] T, [u IN2 v IN2] T), when a., Inliers and affine The relationship with the conversion parameter can be expressed by a linear system shown in the following equation (13).

When this equation (13) is rewritten as R _IN x _IN = s _IN , the least squares solution of the affine transformation parameter x _IN is given by the following equation (14).

In step S56, it outputs the model attitude determined by the affine transformation parameter x _IN as a model recognition result.

  In the above description, the threshold value τ is assumed to be a constant value. However, when performing the iterative processing from step S52 to step S55, first, rough inlier extraction is performed using a relatively large threshold value τ, and the repetition is repeated. A technique such as a so-called “annealing method” that uses a gradually smaller threshold τ each time the number of times increases may be applied. Thereby, an inlier can be extracted with high accuracy.

  Further, in the above description, a pair group P is randomly selected from the candidate corresponding feature point pair group, and the operation of projecting the affine transformation parameters onto the parameter space is repeated, and the cluster having the largest number of members on the parameter space. The affine transformation parameters that determine the model pose by the least squares method were estimated using the elements of, but this is not a limitation. For example, the centroid of the cluster with the largest number of members is used to determine the model pose. It does not matter as a conversion parameter.

  An example of model extraction from an actual object image is shown in FIG. 17A is a model image, FIG. 17B is a feature point extracted from the model image, FIG. 17C is an object image, FIG. 17D is a feature point extracted from the object image, and FIG. (E) shows the final candidate corresponding feature point pair, and FIG. 17 (F) shows the posture of the model in the object image.

(2-2) Object Learning Phase The object learning phase in which the robot apparatus 1 described above autonomously learns an unknown object while moving in the environment will be further described. In this object learning phase, the robot apparatus 1 instructs the user to perform an object learning procedure by voice, and the learning target object is moved in front of the left and right CCD cameras included in the image input apparatus 251 of the robot apparatus 1. Then, the robot apparatus 1 captures an image of the movement of the learning target object, cuts out a model image (object region image described later) of the learning target object from the captured image, and extracts a feature amount (model feature described later) from the model image. (Quantity set) is extracted and registered in the object model database.

  The processing in the object learning phase is shown in the flowchart of FIG. This object learning phase is started when the user speaks a word (hereinafter referred to as “trigger word”) for starting a preset object learning phase.

  First, in step S61, the robot apparatus 1 performs matching between the sound event input from the voice input device 252 and its trigger word in real time, and determines whether or not the trigger word is uttered. Here, when the trigger word is not uttered (No), the process of step S61 is repeated, and when the trigger word is uttered (Yes), the object learning phase is started, and the processes after step S62 are performed.

  Next, in step SS62, the robot apparatus 1 utters the procedure of the object learning phase to the user. In this embodiment, since it is assumed that the user moves the learning target object in front of the CCD camera of the robot apparatus 1, the robot apparatus 1, for example, “ The procedure of object learning is clearly indicated to the user by outputting the voice “move”. Thereby, after this utterance, it is assumed that the user moves the learning target object in front of the CCD camera of the robot apparatus 1.

  Subsequently, in step S63, count values cnt_OK, cnt_NG, cnt_exceed, cnt_close, cnt_fast, cnt_against and cnt_far of a counter described later are initialized to zero.

  Subsequently, in step S64, the robot apparatus 1 extracts a motion region from moving image information (two still images that are temporally adjacent) when the user moves the learning target object. Here, left and right camera images at each time are obtained from the left and right CCD cameras provided in the head unit 3 of the robot apparatus 1. Then, as described above, the distance value of each pixel can be derived from the stereo image data obtained from the left and right camera images. A pixel having a change in luminance value, that is, a difference image is detected from the difference between the current time and the previous camera image. These pixels tend to be detected near the edge of the motion region, and only a part of the motion region is extracted by this operation. Therefore, when a histogram is created with the distance value as the horizontal axis and the number of pixels as the vertical axis for these pixels, the pixel group having the distance d giving a large peak is the motion region. Therefore, in step S64, pixels having a distance value within a predetermined threshold range from d in the previous distance image are extracted, and only a connected region having a large area is extracted as a motion region image.

  The motion region image extracted in step S64 includes a learning target object and a part of a user's hand that moves the learning target object. Therefore, in step S65, the robot apparatus 1 masks the area corresponding to the user's hand and extracts only the object area image. At this time, in this embodiment, the skin color area is regarded as the user's hand, and only the object area image is extracted by masking the skin color area from the motion area image extracted in step S64.

  Here, the skin color area in the image can be extracted using, for example, a technique described in JP-A-2002-157596. Briefly, it is as follows.

  According to Yang et al. (J. Yang, W. Lu, A. Waibel, “Skin-Color Modeling and Adaptation”, Technical Report of CMU, CMU-CS-97-146, 1997) It is experimentally known that on the (r, g) plane obtained by the following formulas (15) and (16) normalized by the formula, the distribution is close to the normal distribution regardless of the person or race. Yes.

Moreover, the above-mentioned Yang et al. Express parameter estimation values of normal distribution representing skin color in this color space as equations (17) and (18) of linear sums of data measured in the past, and weight coefficients α _j , Β _j is derived by maximum likelihood estimation. Here, the equation (17) is an average, and the equation (18) is a variance. Further, m _j and s _j are an average value and a variance value before j steps in the past, respectively. R is the maximum number of histories.

  In the present embodiment, a skin color area of the motion area image is obtained based on this formulation, and the skin color area is regarded as a user's hand and masked.

  An example of operations in steps S64 and S65 is shown in FIG. FIG. 19A shows a camera image by the left CCD camera at the previous time, and FIG. 19B shows a camera image by the left CCD camera at the current time, both taken by the learning object SOb and the user's hand Hd. Has been. FIG. 19C shows range data at the current time, FIG. 19D shows a difference image, and FIG. 19E shows a histogram of distance values related to the difference image. FIG. 19F illustrates a motion region in which a pixel group in the vicinity of a distance that gives a large peak, specifically, a pixel group having a distance value in a range indicated by an arrow in the drawing is extracted from the histogram illustrated in FIG. An image is shown, and FIG. 19G shows the object region image extracted in step S65.

  Subsequently, in step S66, the robot apparatus 1 uses the feature point extraction unit 610a and the feature amount holding unit 620a of the object recognition system (FIG. 9) and the feature points of the object region image and the feature amounts thereof (hereinafter, the feature points and the feature points thereof). Feature values are collectively referred to as “feature values”).

  Subsequently, in step S67, the robot apparatus 1 evaluates the feature amount extraction result. If the extracted feature amount is suitable for generating the feature amount of the model image, the robot device 1 stores it in the time series data memory. The user is instructed how to present the learning target object so that a better feature amount can be extracted. Then, the process proceeds to step S68 (case 1), step S64 (case 2), or step S72 (case 3) according to the evaluation result.

  The processing in step S67 will be described using the flowcharts of FIGS. 20A to 20C. First, in step S81, the robot apparatus 1 determines whether or not the ratio of the pixels overlapping the edge of the image among the pixels on the contour line of the object region obtained in step S65 (FIG. 18) is sufficiently large. Here, when the ratio of the pixels overlapping the edge of the image is not sufficiently large (No), it is considered that the learning target object is within the field of view, and the process proceeds to step S89. On the other hand, when the ratio of the pixels that overlap the edge of the image is sufficiently large (Yes), it is considered that a part of the learning target object is outside the field of view, and the process proceeds to step S82.

  Next, in step S82, the robot apparatus 1 determines whether or not the object area is sufficiently large. If the object area is sufficiently large (Yes), it is determined that the presentation position of the learning target object by the user is too close. In this case, since the extracted feature quantity is inappropriate for model generation, it is not stored in the time-series data memory, and cnt_close is incremented in step S83. In step S84, it is determined whether or not cnt_close exceeds the threshold value. If it does not exceed (No), it is determined as case2 and the process returns to step S64 in FIG. On the other hand, when cnt_close exceeds the threshold (Yes), that is, when the situation where the presentation position of the learning target object is too close has been reproduced more than a certain number of times, in step S85, the user is “moved slightly away”. Then, it is assumed that it is case2, and the process returns to step S64 in FIG. Thereby, it is expected that the user corrects the way of object presentation.

  On the other hand, when the object area is not sufficiently large (No), it is determined that the presentation position of the learning target object by the user is shifted. In this case, since the extracted feature quantity is inappropriate for model generation, it is not stored in the time-series data memory, and cnt_exceed is incremented in step S86. Then, in step S87, it is determined whether or not cnt_exceed exceeds the threshold value. If it does not exceed (No), it is determined as case2 and the process returns to step S64 in FIG. On the other hand, when cnt_exceed exceeds the threshold value (Yes), that is, when the situation where the presentation position of the learning target object is shifted is reproduced a certain number of times or more, in step S88, the user is told “Slightly above (down, After moving “right, left)”, it returns to step S64 in FIG. Thereby, it is expected that the user corrects the way of object presentation.

  When the determination in step S81 is No and the process proceeds to step S89, the robot apparatus 1 determines whether or not the number of feature points extracted from the object region is sufficiently large. Here, if the number of feature points extracted from the object region is not sufficiently large (No), 1) the object is presented in a backlight situation, 2) the object is moved violently, and the result is blurred It can be considered that the object region image has been extracted, or 3) the object presentation position is too far away. Therefore, first, in step S90, it is determined whether or not the light is backlit. At this time, in the present embodiment, a backlight situation is considered when the average gray value of the pixels in the object region is equal to or less than a certain threshold value. For example, the threshold value can be set to 50 when the gray value ranges from 0 (black) to 255 (white). If it is determined in step S90 that the light is backlit (Yes), the extracted feature value is inappropriate for model generation and is not stored in the time-series data memory, and cnt_against is incremented in step S91. Then, in step S92, it is determined whether or not cnt_against exceeds the threshold value. If it does not exceed (No), it is determined as case2 and the process returns to step S64 in FIG. On the other hand, when cnt_against exceeds the threshold value (Yes), that is, when the situation of backlighting is reproduced more than a certain number of times, in step S93, the user utters, “Because of backlighting, move the place and try again” Then, assuming that it is case 3, the process proceeds to step S72 in FIG.

  On the other hand, if it is not backlit in step S90 (No), it is determined in step S94 whether or not the object area is sufficiently large. Here, when the object area is sufficiently large (Yes), it is determined that the learning target object has been presented violently. In this case, since the extracted feature quantity is inappropriate for model generation, it is not stored in the time-series data memory, and cnt_fast is incremented in step S95. Then, in step S96, it is determined whether or not cnt_fast exceeds the threshold value. If it does not exceed (No), it is determined as case2 and the process returns to step S64 in FIG. On the other hand, when cnt_fast exceeds the threshold value (Yes), that is, when the learning target object is moved violently and the situation presented is reproduced more than a certain number of times, in step S97, the user is asked to “move slightly slowly. ”And then returns to step S64 in FIG. 18 assuming that it is case2. Thereby, it is expected that the user corrects the way of object presentation.

  If the object area is not sufficiently large in step S94 (No), it is determined that the position where the learning target object is presented is too far. In this case, since the extracted feature quantity is inappropriate for model generation, it is not stored in the time series data memory, and cnt_far is incremented in step S98. Then, in step S99, it is determined whether or not cnt_far has exceeded the threshold value. If not (No), it is determined to be case 2 and the process returns to step S64 in FIG. On the other hand, when cnt_far exceeds the threshold value (Yes), that is, when the situation where the position to present the learning target object is too far is reproduced more than a certain number of times, in step S100, the user is moved ”And then the process returns to step S64 in FIG. Thereby, it is expected that the user corrects the way of object presentation.

  On the other hand, if a sufficient number of feature points have been extracted in step S89 (Yes), it is determined that the extracted feature values are appropriate for model generation. In step S101, the feature values are stored in the time series memory. save. In step S102, cnt_OK is incremented. In step S103, it is determined whether cnt_OK exceeds a predetermined number N (for example, N = 10). Here, when cnt_OK does not exceed the predetermined number N (No), it is determined as case2 and the process returns to step S64 in FIG. On the other hand, when cnt_OK exceeds the predetermined number N (No), it is determined as case1 and the process proceeds to step S68 in FIG.

  Returning to FIG. 18, after the repetition of step S64 to step S67, the feature amount set of N object region images is held in the time-series data memory. Accordingly, in step S68, the robot apparatus 1 extracts a model feature value set that is a model of the learning target object from the time-series feature value set.

  Here, the extraction of the model feature value set corresponds to the feature value match when the feature value sets at each time in the time series data memory are matched with all combinations, that is, similar to the feature values at other times. This can be realized by selecting only feature quantities that match more from a feature quantity set having more feature quantities that can be taken.

The model feature quantity set extraction algorithm will be described below. Let F ₁ , F ₂ ,..., F _N be feature quantity sets of N object region images in the time-series data memory. For all combinations (F _i , F _j ) where i = 1,..., N, j = 1,..., and N ≠ j, F _i is the feature quantity group of the model image, and F _j is The feature amount comparison unit 630 of the object recognition system (FIG. 9) is assumed to be a feature amount group of the object image, and a candidate corresponding feature point pair group of (F _i , F _j ) is extracted. When the number of candidate corresponding feature point pairs of (F _i , F _j ) is p _ij and the sum of p _ij for all j where j ≠ i is P _i , Max {P _i } is satisfied. Find _i and focus on Fi below.

Specifically, when N = 5, as shown in FIG. 21A, all combinations (F _i , F _j ) where i ≠ j of feature quantity sets F ₁ , F ₂ ,..., F _5. A candidate corresponding feature point pair group is extracted. For example, a candidate corresponding feature point pair group is extracted for the combination of (F ₄ , F ₁ ), (F ₄ , F ₂ ), (F ₄ , F ₃ ), (F ₄ , F ₅ ) for the feature quantity set F _4. To do. As a result, when the number p _ij of candidate corresponding feature point pairs becomes, for example, p ₄₁ = 3, p ₄₂ = 1, p ₄₃ = 2 and p ₄₅ = 2, the sum P ₄ is 8. Here, for example, p ₄₁ = 3 means that there are three candidate corresponding feature point pairs between the feature value set F ₄ and the feature value set F ₁ . Similarly, when P ₁ = 6, P ₂ = 5, P ₃ = 6, and P ₅ = 7, i indicating the maximum value is 4, and attention is paid to F ₄ below.

Next, the feature amount of _{F i v i1, v i2,} ···, and _{v iL.} The number of times each feature quantity v _il (l = 1,..., L) is paired with another feature point is defined as q_v _il, and q_v _il is a sufficiently large feature quantity v _il , for example, the following equation (19) Only the feature quantity v _{il that} satisfies is extracted and set as a model feature quantity set M of the learning target object.

Similarly, when N = 5, as shown in FIG. 21B, the feature quantity v ₄₁ of the feature quantity set F ₄ of interest is the feature quantity feature F ₁ , F ₂ , F ₃ , F ₅ . Since it is paired with a point, q_v ₄₁ = 4. Similarly, when q_v ₄₂ = 2 and q_v ₄₃ = 3, the feature amounts v ₄₁ and v ₄₃ satisfying the above-described equation (15) are set as the model feature amount set M of the learning target object.

  Returning to FIG. 18 again, in step S69, it is determined whether or not the number of feature quantities of the model feature quantity set M is sufficiently large. If the number of features is sufficiently large, the model feature set M is registered in the object model database in step S70, and in step S71, for example, a predetermined sentence such as “object learning is completed” is spoken. Tell the user that object learning has ended. On the other hand, if the number of feature values is not sufficiently large (No) in step S69, the process proceeds to step 72. For example, “Object learning has failed. Learning will be terminated. Tell the user that object learning has failed by speaking the default sentence.

(2-3) Object Recognition Phase The object recognition phase in which the robot apparatus 1 described above recognizes a learned object while moving in the environment will be further described. In this object recognition phase, the feature amount of the input image picked up by the CCD camera provided in the head unit 3 of the robot apparatus 1 is extracted, and the feature amount of the already learned object registered in the object model database is extracted. To detect an object present in the input image in real time.

The process in the object recognition phase is shown in the flowchart of FIG. First, in step S111, the feature amount set F of the input image is extracted using the feature point extraction unit 610b and the feature amount holding unit 620b of the object recognition system (FIG. 9). On the other hand, it is assumed that K model feature value sets M ₁ , M ₂ ,..., M _K corresponding to K learned objects are registered in the object model database.

In step S112, by using the feature quantity comparing unit 630 and the model attitude estimation section 640 of the object recognition system, and matching between feature quantity set F and the model feature quantity set M _i, the object M _i in the input image Detection is performed. At this time, the number of elements of the model feature set M _i, i.e. the number of characteristic quantity and m _i, the number of elements taken correspond wherein the amount set F of feature amounts set M _i as mf _i, the following equation (20 as shown in), to define the ratio of _{m i} of mf _i and confidence _{C i} of the detection result object _{M i.}

In step S113, the recognition result of the learned object in the input image is output. At this time, the recognition result of each existing learning object, a confidence C _i, the estimated posture (affine transformation parameters in the case of mf _i is 3 or more in the input image already learning object: translation amount already learning object, (Rotation angle, magnification).

In this method, since it is subjected to matching by independent model feature quantity set M _i for each already learning object, it is possible to cope with a case where there are a plurality of previously learned object in the input image. Further, the certainty factor C _i can be used, for example, for situation determination when a plurality of objects are detected in an overlapping manner. For example, if two objects A and B are both detected with high certainty, it can be determined that one of them partially hides the other. On the other hand, when only A has a high certainty factor, it can be estimated that most of B is hidden by A or that B is likely to be a false detection. Further, when both A and B have low certainty, it can be estimated that any object is highly likely to be erroneously detected. By outputting the fuzzy recognition result in this way, it becomes possible to give flexibility to the action selection of the robot apparatus 1 performed based on the recognition result.

(3) Modifications The best mode for carrying out the present invention has been described above. However, the present invention is not limited to the above-described embodiment, and various modifications can be made without departing from the gist of the present invention. Of course, it can be changed.

  For example, in the above-described embodiment, the case where the present invention is applied to a bipedal walking robot apparatus has been described. However, the present invention is not limited to this, and other various robot apparatuses and other apparatuses other than the robot apparatus may be used. Can be widely applied to.

  In the embodiment described above, the object learning phase is started by recognizing a predetermined trigger word. However, the present invention is not limited to this. For example, a specific switch provided in the robot apparatus is used. The object learning phase may be started by using a sensor input such as as a trigger. This technique is effective when the present invention is applied to a robot apparatus having no voice recognition means.

  In the above-described embodiment, the difference image and the distance information are used when extracting the moving region. However, the present invention is not limited to this. For example, a technique using an optical flow (“Okada, See Shirai, Miura, and Kuno, "Tracking of moving objects based on optical flow and distance information", IEICE Transactions, D-II, Vol.J80-D-II, No.6, pp.1530-1538, 1997. ) And a method using the watershed method (“Sakaida, Naemura, Kinji,“ Extraction of moving image objects using the background difference method and the region growth method by spatio-temporal watershed ”), IEICE Transactions D-II, Vol.J84-D-II, No.12, pp.2541-2555, 2001 ") can be applied.

  Further, in the above-described embodiment, the case where the technique proposed in the specification and drawings of Patent Application 2003-124225 is used as a basic object recognition technology has been described. An object recognition system can be used. In this case, the feature model set used by the object recognition system may be held in the object model database. Further, when using an object recognition system that does not output a candidate corresponding feature point pair group, the present invention can be adapted by repeating step S64 to step S67 of FIG. 18 only once.

  In the above-described embodiment, the end of the object learning phase has been described as being clearly shown to the user by synthesizing and speaking a predetermined sentence. However, the present invention is not limited to this and is installed in the robot apparatus. The end of the object learning phase may be clearly indicated to the user by emitting a specific LED (Light Emission Diode) or outputting a specific sound from a speaker. This technique is effective when the present invention is applied to a robot apparatus that does not have a speech synthesis function.

  According to the above-described present invention, when the robot apparatus learns the visual characteristics of an object existing in the external environment in which the robot apparatus acts, the user first makes a voice call so that the learning target object to be learned is moved. By giving an instruction, capturing an image of the user moving the learning target object, extracting the object region image of the learning target object from the captured image, and extracting and holding a feature amount group from the object region image, Even in a real environment where the background changes variously, it is possible to autonomously learn objects in the environment with high accuracy. In particular, since an object can be learned through dialogue, the burden on the user who learns the object is reduced, and the entertainment characteristics of the robot apparatus are also improved. In addition, a feature amount group is extracted from the input image captured by the image pickup unit of the robot apparatus, and the feature amount group is compared with the feature amount group of the learned object stored in advance, so that it has been learned in the input image. Whether or not an object exists can be detected in real time.

It is a perspective view which shows the external appearance structure of the robot apparatus in this Embodiment. It is a figure which shows typically the freedom degree structural model of the robot apparatus. It is a figure which shows typically the control system structure of the robot apparatus. It is a functional block diagram which shows the part regarding the voice interactive function of the robot apparatus. It is a figure which shows typically the part regarding the autonomous action function of the robot apparatus. It is a flowchart explaining operation | movement of the software of the robot apparatus. It is a flowchart explaining the calculation / update process of the grid occupancy rate of an obstruction, and the calculation / update process of an absolute coordinate. It is a figure which shows an example of the environmental map which the robot apparatus produces. It is a figure which shows schematic structure of the object recognition system in this Embodiment. It is a figure explaining the construction method of the multi-resolution pyramid structure of the image in the feature point extraction part of the object recognition system. It is a flowchart explaining the detection process of the feature point which does not change a position by the resolution change to the L level. It is a figure which shows the example in the case of detecting the feature point which does not change a position by the resolution change to the 3rd level. It is a figure explaining the process in the feature-value holding | maintenance part of the same object recognition system. FIG. 5B shows an example of a gradient direction histogram obtained from the density gradient information of FIG. It is a flowchart explaining the detail of the process in the feature-value comparison part of the same object recognition system. It is a figure explaining the calculation method of the similarity between density gradient vector _Um , _Uo . It is a flowchart explaining the detail of the process in the model attitude | position estimation part of the same object recognition system. It is a figure which shows a mode that a model is extracted from an actual object image. It is a flowchart explaining the process in the object learning phase of the robot apparatus. It is a figure which shows a mode that a motion area image and an object area | region image are extracted from an actual image. It is a flowchart explaining the evaluation method of a feature-value extraction result. It is a flowchart explaining the evaluation method of a feature-value extraction result. It is a flowchart explaining the evaluation method of a feature-value extraction result. It is a figure which shows the specific example which extracts the model feature-value set of a learning target object from a time series feature-value set. It is a flowchart explaining the process in the object recognition phase of the robot apparatus.

Explanation of symbols

  1 robot apparatus, 2 trunk unit, 3 head unit, 4R / L arm unit, 5R / L leg unit, 400 speech recognition unit, 410 response generation unit, 410a, 410b ... response generation module, 420 dialogue Control unit, 430 Speech synthesis unit, 600 Object recognition system, 610a, 610b Feature point extraction unit, 620a, 620b Feature quantity holding unit, 630 Feature quantity comparison unit, 640 Model posture estimation unit

Claims (22)

In a robotic device that learns the visual features of objects in the environment,
Voice recognition means for recognizing voice and determining whether or not an instruction to start object learning is given by voice from the user;
If it is determined by the voice recognition means that the user has given an instruction to start object learning, the user is instructed to present the object to be learned while moving as a procedure for object learning. Audio output means for outputting audio;
Imaging means for imaging the learning object;
Image extracting means for extracting an object region image occupied by the learning target object in the captured image;
Feature quantity holding means for extracting and holding a feature quantity group consisting of a plurality of feature quantities from the object region image extracted by the image extraction means ,
The feature amount holding unit determines whether or not the learning target object by the user is presented in an appropriate situation by evaluating a result of feature amount extraction from the object region image, and an inappropriate situation If it is determined that it is presented below, the speech output means does not hold the feature quantity group extracted from the object region image, and the sound output means A robot device that outputs a voice instructing re-presentation . The image extracting unit extracts a motion region image is an area for which the change before and after the captured image moving said learning object from the motion region image according to claim 1, wherein you extract the object area image Robot device. The image extraction means, and masking a region corresponding to the hand of the user moves the learning target object from the moving area image, a robot apparatus according to claim 2, wherein that to extract the object area image. The image extracting means, a robot apparatus according to claim 3, wherein Ru Masukingusu skin color region in the motion region image. The appropriate and non situation, backlit, too close, too far, or robotic device context der Ru claim 1, wherein the off-field. The image extraction means extracts a time-series object region image from a time-series captured image obtained by repeatedly capturing the learning target object by the imaging means,
The feature amount holding means extracts a feature amount group from each of the time-series object region images, and selects a feature amount of the learning target object from all the extracted feature amount groups of the time-series object region image. robot apparatus according to claim 1, wherein you. The feature amount holding means includes a feature amount group having more feature amounts similar to the feature amounts included in the feature amount group of the object region image at different times among all the feature amount groups of the time-series object region image. selected, the chosen from among the feature amount group, a robot apparatus according to claim 6, wherein you select only the feature amount that is similar to the threshold value or more feature quantity. Said voice output means, said when the feature quantity retention means feature amount group of the learning object is held, the robot apparatus according to claim 1, wherein you outputs sound for ending the object learning with respect to the user . Feature quantity extraction means for extracting a feature quantity group from an input image captured by the imaging means;
The feature quantity group of the learned object held in the feature quantity holding means is compared with the feature quantity group extracted by the feature quantity extraction means, and the learned object exists in the input image. robot apparatus further comprising Ru claim 1, wherein the feature quantity comparison means for detecting whether. As a result of the comparison by the feature amount comparison unit, when a feature amount similar to the feature amount included in the feature amount group of the input image exists among the feature amounts included in the feature amount group of the learned object, , the feature quantity comparison means, claim the similar feature quantity is a percentage of the characteristic amount group of the learned object, you output as the reliability of the learned object present in the input image 9 The robot apparatus described. As a result of the comparison by the feature amount comparison means, there are three or more feature amounts similar to the feature amounts included in the feature amount group of the input image among the feature amounts included in the feature amount group of the learned object. when the robot apparatus further comprises Ru claim 9, wherein the posture estimating means for estimating the position and orientation of the learned object during the input image. In an object learning method of a robot apparatus that learns visual features of an object existing in an environment,
A voice recognition step of recognizing voice and determining whether or not an instruction to start object learning is given by voice from the user;
When it is determined in the voice recognition step that the user has given an instruction to start object learning, the user is instructed to present the learning target object to be learned while moving as an object learning procedure. A presentation instruction step for outputting a sound to be performed;
An imaging step of imaging the learning target object by an imaging means;
An image extraction step of extracting an object region image occupied by the learning target object in the captured image;
Possess from the object area image extracted by the image extraction step, the feature amount holding step of holding by extracting a feature amount group including a plurality of feature amounts,
In the feature amount holding step, it is determined whether or not the learning target object by the user is presented under an appropriate situation by evaluating a result of feature quantity extraction from the object region image. If it is determined that it is presented below, the speech output means does not hold the feature quantity group extracted from the object region image, and the sound output means An object learning method for outputting a voice instructing re-presentation . In the image extracting step, from the captured image to extract the moving area image is a region of change before and after moving the learning target object from the motion region image according to claim 12, wherein that to extract the object area image Object learning method. Above the image extraction process, by masking the region corresponding to the hand of the user from the moving area image has moved the learning target object, the object learning method according to claim 13, wherein that to extract the object area image. Above the image extraction step, the object learning method according to claim 14, wherein Ru Masukingusu skin color region in the motion region image. The appropriate and non situation, backlit, too close, too far, or object learning method according to claim 12, wherein Ru situation der deviated from the field of view. In the image extraction step, a time-series object region image is extracted from a time-series captured image obtained by repeatedly capturing the learning target object by the imaging means,
In the feature amount holding step, a feature amount group is extracted from each of the time-series object region images, and a feature amount of the learning target object is selected from all the feature amount groups of the extracted time-series object region images. object learning method according to claim 12, wherein you. In the feature amount holding step, a feature amount group having more feature amounts similar to the feature amounts included in the feature amount group of the object region image at different times among all the feature amount groups of the time-series object region image. selected, the chosen from among the feature amount group, the object learning method according to claim 17, wherein you select only the feature amount that is similar to the threshold value or more feature quantity. When the feature quantity group of the learning object are held in the feature holder step, according to claim 12, wherein that further having a termination instruction step of outputting a sound for ending the object learning with respect to the user Object learning method. A feature amount extraction step of extracting a feature amount group from the input image imaged by the imaging means;
The feature quantity group of the learned object held in the feature quantity holding step is compared with the feature quantity group extracted in the feature quantity extraction step, and the learned object exists in the input image. whether the feature quantity comparison step and the further object learning method according to claim 12, wherein that having a for detecting the to. As a result of the comparison in the feature amount comparison step, when there is a feature amount similar to the feature amount included in the feature amount group of the input image among the feature amounts included in the feature amount group of the learned object in the feature quantity comparison step, claim the similar feature quantity is a percentage of the characteristic amount group of the learned object, you output as the reliability of the learned object is present in the input image 20 The object learning method described. As a result of the comparison in the feature amount comparison step, among the feature amounts included in the feature amount group of the learned object, there are three or more feature amounts similar to the feature amounts included in the feature amount group of the input image. case, further object learning method according to claim 20, wherein that having a pose estimation process for estimating the position and orientation of the learned object during the input image.

Images