JP5068711B2 - Object shape recognition system and object shape recognition method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To easily recognize the shape of an object. <P>SOLUTION: This device 10 for recognizing an object shape includes a camera 11 for imaging the object whose shape is to be recognized, a microphone 12 and sound detecting section 13 for detecting sound generated from the object, a position detecting section 14 for detecting the position at which the sound is generated in an image imaged by the camera 11 at a timing at which the sound is detected, and a shape estimating section 16 for estimating the shape of the object from the detected position. <P>COPYRIGHT: (C)2010,JPO&amp;INPIT

Description

  The present invention relates to an object shape recognition system and an object shape recognition method for recognizing the shape of an object.

  When projecting images or superimposing them on an actual object (Augmented Reality (AR), Mixed Reality (MR)) with a projector or HMD (head-mounted display), it is usually decided beforehand such as a screen or a predetermined desk. Projects and superimposes an image on a specified range or object.

  However, if an image can be projected and superimposed only on a predetermined object or range, the place where it can be used is limited. If an object or range whose shape is not known is recognized, and an image can be projected and superimposed on the object or range, the image can be projected and superimposed anytime and anywhere.

There are the following methods for recognizing the shape of an object. For example, Patent Document 1 discloses a method for recognizing a three-dimensional shape of an object using a distance image. Patent Document 2 discloses a method of capturing an image of a background in advance, and Patent Document 3 illustrates a method of creating a model of an object to be recognized.
Japanese Patent Laid-Open No. 2-181880 JP 2007-26327 A JP 2007-42136 A

  However, as known as the above-mentioned technique, it is very difficult to specify only the shape of an object whose shape is not known in advance. For example, with the technique described in Patent Document 1, even if the shape of an object can be recognized, the object to be recognized cannot be specified. In addition, in the techniques described in Patent Documents 2 and 3, prior preparation is required before the object is recognized. For example, the shape of the object and the situation (background) in which the object is used must be known in advance. .

  The present invention has been made in view of the above situation, and an object thereof is to provide an object shape recognition system and an object shape recognition method that can easily recognize the shape of an object. Here, the shape of the object to be recognized includes a partial range of the object on which the video as described above is projected and superimposed.

  In order to achieve the above object, an object shape recognition system according to the present invention includes an imaging unit that images an object whose shape is to be recognized, a sound detection unit that detects a predetermined sound, and a sound that is detected by the sound detection unit. A position detection unit that detects a position corresponding to the shape of the object in the image captured by the imaging unit at a specified timing, and a shape estimation unit that estimates the shape of the object from the position detected by the position detection unit. It is characterized by that.

  When an object shape is recognized in the object shape recognition system according to the present invention, a sound is generated by a user or the like. On the other hand, the object is imaged at the timing when the sound is generated, and a position corresponding to the shape of the object in the captured image is detected. The shape of the object is specified from the detected position. As described above, in the object shape recognition system according to the present invention, the object shape is estimated by imaging an object, detecting sound, and detecting a position corresponding to the object shape. These can be performed more easily than conventional shape recognition. As a result, the object shape recognition system according to the present invention can easily recognize the shape of the object.

  It is desirable that the sound detection unit detects a sound generated from the object as a predetermined sound, and the position detection unit detects a position where the sound is generated as a position corresponding to the shape of the object. In this configuration, sound is generated from a predetermined position of an object whose shape is to be recognized by a user or the like. On the other hand, the object is imaged at the timing when the sound is generated, and the position where the sound is generated in the captured image is detected. The shape of the object is specified from the detected position. That is, according to this configuration, the shape of the object is estimated by imaging the object, detecting the sound, and detecting the position where the sound is generated. As a result, according to this configuration, the shape of the object can be recognized more easily.

  The sound generated from the object is the sound of the object being struck, and the position detection means stores in advance information related to the object that is struck, and the position where the object is struck based on the information is determined. It is desirable to detect it as a position where the occurrence occurs. Sound can be easily and reliably generated by hitting an object whose shape is to be recognized. In addition, by determining why the object is struck, the position where the sound is generated is surely detected by the above configuration. That is, according to the above configuration, the shape of the object can be recognized more easily and reliably.

  It is desirable that the sound detection unit stores information related to the sound to be detected in advance and detects a predetermined sound based on the information. According to this configuration, it is possible to reliably detect a predetermined sound and to reliably recognize the shape of the object.

  It is desirable that the position detection unit detects positions corresponding to the shapes of the plurality of objects, and the shape estimation unit estimates the shapes of the objects from the plurality of positions detected by the position detection unit. According to this configuration, since the shape of the object is estimated from a plurality of positions, it is possible to appropriately recognize the shape of the object.

  The imaging means further includes a position tracking means for capturing an object over a plurality of times and detecting a position corresponding to the position detected by the position detection means in the time-changed image captured by the imaging means. It is desirable to estimate the shape of the object from the position detected by the position tracking means. According to this configuration, the detected position is tracked. For example, even if the object itself or the imaging direction moves while detecting a plurality of positions, the shape of the object can be estimated based on an appropriate position. it can. That is, according to the above configuration, the shape of the object can be recognized more appropriately.

  The object shape recognition system preferably further includes projection means for projecting an image in accordance with the shape of the object estimated by the shape estimation means. According to this configuration, an image can be projected on the recognized shape, and the above-described projection and superimposition can be appropriately performed.

  The imaging means further includes shape follow-up means for imaging an object over a plurality of times, and detecting a shape corresponding to the shape of the object estimated by the shape estimation means in the time-changed image captured by the imaging means. The means desirably projects an image in accordance with the shape detected by the position following means. According to this configuration, even if the object itself or the imaging direction moves, the above-described projection and superimposition can be appropriately performed.

  By the way, the present invention can be described as an invention of an object shape recognition system as described above, and can also be described as an invention of an object shape recognition method as follows. This is substantially the same invention only in different categories, and has the same operations and effects.

  That is, the object shape recognition method according to the present invention includes an imaging step of imaging an object whose shape is to be recognized, a sound detection step of detecting a predetermined sound, and an imaging step at a timing when sound is detected in the sound detection step. A position detecting step for detecting a position corresponding to the shape of the object in the image picked up in (2), and a shape estimating step for estimating the shape of the object from the position detected in the position detecting step.

  In the present invention, the shape of the object is estimated by imaging the object whose shape is to be recognized, detecting the sound, and detecting the position where the sound is emitted. These can be performed more easily than conventional shape recognition. As a result, according to the present invention, the shape of an object can be easily recognized.

  Hereinafter, preferred embodiments of an object shape recognition system and an object shape recognition method according to the present invention will be described in detail with reference to the drawings. In the description of the drawings, the same elements are denoted by the same reference numerals, and redundant description is omitted.

  FIG. 1 schematically shows an external configuration of an object shape recognition apparatus 10 that is an embodiment of an object shape recognition system according to the present invention. The object shape recognition device 10 is a device for recognizing the shape of an object. In the present embodiment, specifically, the object shape recognition device 10 has the following functions. As shown in FIG. 1, the object shape recognition apparatus 10 is a glasses-type display and has a shape that can be worn by a user. The object shape recognition apparatus 10 has a function of a computer input device. Specifically, the object shape recognition apparatus 10 projects an image 30 of an input device such as a (virtual) keyboard or a touch panel so as to be superimposed on a predetermined object 20, and the input device is presented to the user. Recognize the video of the video. The object shape recognizing device 10 detects an operation with a finger or a pen on the video (corresponding to the user) by the user and uses it as an input to the computer.

  The predetermined object 20 on which the video 30 of the input device is superimposed preferably has a flat surface, for example, a notebook or notebook held by the user or a fixed wall. Further, the image 30 of the input device may be superimposed on the object 20 by being projected. Further, when the glasses-type display is of an optical see-through type, only the image 30 of the input device is seen so as to be superimposed on the lens of the glasses-type display when the user views the object 20 through the lens. It may be projected. When the eyeglass-type display is a video see-through type, the object 20 captured by the camera may be simultaneously projected onto the lens. At this time, the same image may be projected to both eyes, or an image viewed from each eye may be estimated from the distance between the right eye and the left eye, and the images projected to each eye may be prepared separately. When different videos are projected according to each eye, it is possible to show the user a stereoscopic image (depth).

  Here, the user specifies the shape (or range) of the object 20 onto which the video 30 is projected. As shown in FIG. 2, this designation is performed, for example, by making a sound by hitting a corner (feature point) 21 of the shape of the object 20 with a finger. The shape of the designated object 20 is recognized by the object shape recognition apparatus 10. That is, the shape recognition target in the present embodiment is the object 20 on which the video 30 is superimposed. The shape recognition will be described in detail later.

  Subsequently, the function of the object shape recognition apparatus 10 will be described. As shown in FIG. 3, the object shape recognition apparatus 10 includes a camera 11, a microphone 12, a sound detection unit 13, a position detection unit 14, a position tracking unit 15, a shape estimation unit 16, and a shape tracking unit 17. And a display 18 and a video storage unit 19.

  The camera 11 is an imaging unit that captures an image of the object 20 whose shape is to be recognized. The camera 11 images the object 20 over a plurality of times. That is, the camera 11 images the object 20 as a moving image. The camera 11 is provided so that its imaging direction is the same as the direction of the user's line of sight, that is, the optical axis direction of the lens of the glasses-type display. Accordingly, the camera 11 captures an image of the object 20 when the user directs his / her line of sight toward the object 20 (when the user directs his / her line of sight toward the object 20, the imaging direction becomes the direction toward the object 20). The camera 11 may be provided integrally with the eyeglass-type display, or may be provided as a separate body around the camera 11 or carried by the user. The camera 11 outputs the captured image (data) to the position detection unit 14. In addition, the camera 11 outputs the image to the position follower 15 and the shape follower 17 in order to follow the position and shape to be described later.

  The microphone 12 is a function of sound detection means for detecting sound generated from the object 20. The microphone 12 collects sound generated around the object 20 including the object 20. The microphone 12 outputs the collected sound (data) to the sound detection unit 13.

  The sound detection unit 13 is a function of sound detection means for detecting that the sound collected by the microphone 12 includes a specific sound generated from the object 20. The specific sound generated from the object 20 is a sound that the object 20 is hit by the user. Specifically, the sound detection unit 13 stores in advance information related to the sound of the object 20 being struck, for example, a sound pattern (rhythm, voice, volume), and inputs the sound pattern and the microphone 12. The above detection is performed by judging whether or not the sound that is made matches (there is a part). That is, the sound detection unit 13 stores (registers) a sound to be detected in advance, and detects the stored sound as a sound generated from the object 20. This registration may be performed by the developer of the object shape recognition apparatus 10, or the user may register a desired pattern by himself / herself. When a specific sound is detected, the sound detection unit 13 notifies the position detection unit 14 to that effect.

  The position detection unit 14 detects a position (feature point) where the specific sound is generated in an image captured by the camera 11 at the timing when the specific sound is detected by the sound detection unit 13. It is. Specifically, the position detection unit 14 detects the position from the image input from the camera 11 at the timing when the sound detection unit 13 notifies that the specific sound has been detected. The position detection unit 14 stores in advance information related to what strikes the object 20, and detects the position (position where the object 20 is hit) of the object 20 based on the information from the above timing image. The detected position is the position where the specific sound is generated. What strikes the object 20 is, for example, a tool such as a user's finger or a stick used by the user, and is designated in advance.

  Detection of the position in the image of an object 20 such as a user's finger is performed by image processing. Specifically, for example, the position detection unit 14 stores in advance the skin color that is the color of the finger as information related to the object 20 hitting, and the largest area (the number of pixels is large) among the skin color areas in the image. (Region) is detected and the uppermost coordinate is detected. Alternatively, the unevenness of the skin color region may be detected, a specific shape (for example, an ellipse) stored in advance by the position detection unit 14 may be recognized, and a predetermined point of the shape may be detected. Alternatively, a hand model may be created and stored in the position detection unit 14 in advance, and the fingertip may be detected based on the information, and the fingertip point may be detected. The position detection unit 14 outputs (two-dimensional) coordinate data of the position detected as described above to the position tracking unit 15 and the shape estimation unit 16 in the image. The position detection is performed a plurality of times (for example, three times) for the detection of the shape of the (one) object 20. The greater the number of times, the more detailed the shape of the object 20 can be recognized.

  The position detection unit 14 is notified of the timing of detecting the position from the image from the sound detection unit 13, but the notification is not necessarily performed. For example, information on the time taken is associated with an image picked up by the camera 11, and a notification of the time when the specific sound is detected is received from the sound detection unit 13, and the position is determined based on the time. The image to be detected may be specified.

  The position follower 15 is a position follower that detects (follows) a position corresponding to the position detected by the position detector 14 in the image captured by the camera 11. Imaging by the camera 11 is performed while a plurality of position detections are performed by the position detection unit 14, and the captured image changes with time. If the object 20 is held by hand before the next point is detected after the first position is detected, the position of the detected point in the image may move. Further, the imaging direction of the camera 11 may move, and the position of the detected point in the image may move. The position detection (following) in the position tracking unit 15 is for appropriately estimating the shape when the position is specified using images over a plurality of times as described above.

  Specifically, the position tracking unit 15 extracts an image in a predetermined range near the coordinates detected by the position detection unit 14 from the image used for detection by the position detection unit 14. The position follower 15 stores the image in the predetermined range as an image indicating the feature of the detected position. Subsequently, the position tracking unit 15 detects a portion corresponding to the stored image in the predetermined range from the image to be tracked, and specifies a position (two-dimensional coordinates on the image) tracked from the portion. . Specifically, this detection is performed using, for example, a feature point tracking method using an optical flow or the like. The position tracking unit 15 outputs the coordinate data of the tracked position to the shape estimation unit 16.

  The shape estimation unit 16 is a shape estimation unit that estimates the shape of the object 20 from the position detected by the position detection unit 14 and the position tracked by the position tracking unit 15. The shape estimated here is a two-dimensional shape (a shape in an image captured by the camera 11). The estimation of the shape here includes estimating the position of the shape (in the image captured by the camera 11). The shape estimation unit 16 performs shape estimation when the position detected by the position detection unit 14 (followed by the position tracking unit 15) reaches a predetermined number (for example, three points). The number of positions used for shape estimation is stored in the shape estimation unit 16 in advance.

  Specifically, for example, the shape estimation unit 16 estimates the shape of the object 20 by connecting the detected positions. In addition, the shape estimation unit 16 may be a shape that infers a graphic (for example, a quadrangle) that is inscribed in the graphic having a shape that connects the detected positions, or a graphic that includes all the detected positions. Alternatively, the size of the projection range may be set in advance, and the shape may be estimated by approximating the detected position within the range. The shape estimation unit 16 stores information for estimating the shape (for example, information when a shape such as a rectangle is determined) or a rule in advance, and estimates the shape using the information or the rule. . The shape estimation unit 16 outputs information indicating the estimated shape to the shape tracking unit 17 and the display 18. Here, the information indicating the estimated shape includes information indicating the position of the shape in the image.

  The shape follower 17 is a shape follower that detects (follows) a shape corresponding to the shape of the object 20 estimated by the shape estimator 16 in the image captured by the camera. Imaging by the camera 11 is performed while a video is projected on the display 18 described later, and the captured image changes with time. The detected shape in the image captured as described above can be changed by the movement of the object 20 or the camera 11. The projection of the image by the display 18 is performed in accordance with the detected shape of the object 20, and is intended to appropriately project the image when the image is projected over a plurality of times. .

  Specifically, the shape follower 17 acquires information indicating the feature of the shape from the image at the time when the shape is estimated by the shape estimator 16. For example, an image of the shape range estimated by the shape estimation unit 16 is extracted. The shape following unit 17 stores the image in the predetermined range as an image (template) indicating the detected feature of the shape. Subsequently, the shape tracking unit 17 detects a portion corresponding to the stored image (template) of the predetermined range from the image to be tracked, and detects the portion as the shape of the object 20 in the image to be tracked. To do. Specifically, this detection is performed using, for example, a pattern matching (template matching) method or the like. The position tracking unit 15 outputs information indicating the tracked shape to the display 18.

  The information indicating the feature of the shape to be compared with the image to be followed does not have to be the image itself. For example, the color information (histogram or average color) of the image in the extracted range may be used as information indicating the characteristics of the shape, and the region having the same color information of the image to be followed may be followed. Further, information indicating the characteristics of the edge (end) of the object 20 may be used as a template. Similarly to the position follower 15 described above, all the positions detected by the position detector 14 are tracked, and the shape is estimated from the tracked position in the same manner as the shape estimator 16 to follow the shape. It may be done. The shape tracking unit 17 outputs information indicating the tracked shape to the display 18. Here, the information indicating the following shape includes information indicating the position of the shape in the image.

  The display 18 projects the image 30 according to the shape of the object 20 estimated by the shape estimation unit 16 and the shape of the object 20 tracked by the shape tracking unit 17 (hereinafter referred to as recognition shapes). It is. The display 18 acquires the image to be projected from the image storage unit 19 and performs projection. As described above, the display 18 is provided, for example, on a lens of a glasses-type display, and projection of an image is performed so as to be superimposed on the object 20 as described above. Based on the information input from the shape estimation unit 16 or the shape tracking unit 17, the display 18 converts the video acquired from the video storage unit 19 to match the recognition shape, and projects the converted video 30. For example, the images to be projected stored in the video storage unit 19 are projected with the same size characters arranged in a rectangular range as shown in FIG. 4A, and the recognition shape is trapezoidal. If so, the conversion is performed so that the character on the short side (left side) side becomes smaller than the character on the long side (right side) side, as shown in FIG. In addition, the display 18 is based on the information input from the shape estimation unit 16 or the shape tracking unit 17, and the video 30 is displayed at a location where the object 20 is located in the image captured by the camera 11 (so that the user can visually recognize the image). Project.

上記の式において、Ｒ_１ｘ，Ｒ_２ｘ，Ｒ_３ｘ，Ｒ_１ｙ，Ｒ_２ｙ，Ｒ_３ｙ，Ｒ_１ｚ，Ｒ_２ｚ，Ｒ_３ｚは回転パラメータであり、ΔＸ，ΔＹ，ΔＺは平行移動パラメータである。変換行例Ｍは、物体２０の認識形状の座標（ｘ，ｙ，ｚ）に合わせて、投影すべき画像の座標（Ｘ，Ｙ，Ｚ）をそれぞれの軸に対して回転移動及び平行移動させるための行列である。ここで、認識形状及び投影すべき画像は二次元であるのでｚ＝Ｚ＝０である。ディスプレイ１８が、これらのパラメータを、認識形状及び映像蓄積部１９から取得した映像の形状とそれぞれの位置を示す情報とから算出して変換を行う。認識形状が回転している場合は、回転軸毎に回転パラメータＲ部分が、認識形状の回転角度θａから求めることができる。平行移動認識形状が平行移動している場合は、平行移動分ΔＸ，ΔＹ，ΔＺのうちその移動軸に沿ったパラメータが設定される。 This conversion and alignment is realized by the display 18 performing an existing image conversion process for enlarging, reducing, rotating and translating the video acquired from the video storage unit 19. For example, the image conversion process is performed by converting the video (data) acquired from the video storage unit 19 in the following conversion example M.

In the above formula, R _1x , R _2x , R _3x , R _1y , R _2y , R _3y , R _1z , R _2z , R _3z are rotational parameters, and ΔX, ΔY, ΔZ are parallel movement parameters. In the conversion row example M, the coordinates (X, Y, Z) of the image to be projected are rotated and translated with respect to the respective axes in accordance with the coordinates (x, y, z) of the recognition shape of the object 20. Is a matrix for Here, since the recognition shape and the image to be projected are two-dimensional, z = Z = 0. The display 18 performs conversion by calculating these parameters from the recognized shape and the shape of the video acquired from the video storage unit 19 and information indicating the respective positions. When the recognition shape is rotating, the rotation parameter R portion can be obtained for each rotation axis from the rotation angle θa of the recognition shape. When the parallel movement recognition shape is moving in parallel, a parameter along the movement axis is set among the parallel movements ΔX, ΔY, and ΔZ.

  The display 18 performs a conversion process on the video as described above and projects it. By doing so, the superimposed image 30 is tilted in accordance with the tilt of the object 20 as shown in FIG.

  The video storage unit 19 stores video (data) 30 projected by the display 18 and outputs the video to the display 18 in accordance with a request from the display 18.

  The object shape recognition apparatus 10 detects an input operation performed by a user on (an image corresponding to) an image such as a (virtual) keyboard or touch panel projected by the display 18 and uses the input operation as input information. Etc. (not shown). This recognition can also be performed by, for example, recognizing the sound and finger position as described above using an image captured by the camera 11. The detection of the finger position only needs to be performed within the range of the shape (detected or followed) of the object 20 in the image by the camera 11 and can be detected faster and more accurately than searching for the finger from the entire image. it can. The above is the function of the object shape recognition apparatus 10.

  As shown in FIG. 5, an object shape recognition apparatus 10 includes a central processing unit (CPU) 101, a random access memory (RAM) 102 and a read only memory (ROM) 103 that are main storage devices, and an auxiliary storage device such as a hard disk. A computer having hardware such as 104 is configured. As other hardware, the object shape recognition apparatus 10 includes the camera 11, the microphone 12, and the display 18 described above. When these components operate, the above-described functions of the object shape recognition apparatus 10 are exhibited.

  Subsequently, processing (object shape recognition method) executed by the object shape recognition apparatus 10 according to the present embodiment will be described using the flowchart of FIG. This process is performed when the user uses the function of the input device of the object shape recognition apparatus 10 described above. The process starts when the user wears the object shape recognition device 10 and performs an operation to start the function with respect to the object shape recognition device 10.

  First, in the object shape recognition device 10, the camera 11 starts imaging the object 20 as a shape recognition target (S01, imaging step). At this time, when the user wearing the object shape recognition device 10 faces the object 20, the imaging direction of the camera 11 becomes the direction of the object 20. This imaging is continuously performed during this processing. The captured image is output to the position detection unit 14, the position tracking unit 15, and the shape tracking unit 17 every time it is captured.

  Subsequently, the user hits the object 20 with a preset object such as a finger. The location of the object 20 hit by the user is a position where the shape of the object 20 can be recognized, such as the corner 21 of the object 20 as described above. When the object 20 is hit by the user, in the object shape recognition device 10, the sound is collected by the microphone 12, and the sound is input from the microphone 12 to the sound detection unit 13. Subsequently, the sound that the object 20 is hit is detected by the sound detection unit 13 (S02, sound detection step). When sound is detected, the sound detection unit 13 notifies the position detection unit 14 to that effect.

  When the sound detection unit 13 notifies the position detection unit 14 that sound has been detected, the position detection unit 14 detects the position (feature point) where the sound is generated in the image captured by the camera (S03). , Position detection step). Information indicating the detected feature points is output from the position detection unit 14 to the position tracking unit 15 and the shape estimation unit 16. Here, the process branches as follows depending on whether or not the number of detected feature points has reached 3, which is the number necessary for shape estimation (S04).

  When the number of detected feature points is less than 3, the position tracking unit 15 tracks the position corresponding to the detected position in the image by the camera 11 (S05, position tracking step). Information indicating the tracked position is output from the position tracking unit 15 to the shape estimation unit 16. The tracking of the position is continuously performed until the number of detected feature points becomes three. In addition, the sound detection (S02) and the position detection (S03) are performed.

  On the other hand, after the process of S03, when the number of detected feature points is 3 or more, the shape of the object 20 is estimated by the shape estimation unit 16 (S06, shape estimation step). Information indicating the estimated shape is output from the shape estimation unit 16 to the shape tracking unit 17 and the display 18. Based on the information indicating the estimated shape for tracking the estimated shape, the shape tracking unit 17 acquires and stores information indicating the feature of the shape (S07, shape tracking step).

  Subsequently, the projected image is acquired from the image storage unit 19 by the display 18. Subsequently, on the basis of the information indicating the shape of the object 20 input from the shape estimation unit 16 and the shape tracking unit 17, the display 18 calculates parameters for converting the video (S08, projection). Step). Subsequently, the display 18 performs conversion processing of an image to be projected using the calculated parameters (S09, projection step). Subsequently, the converted video 30 is projected by the display 18 so as to be superimposed on the object 20 as described above (S10, projection step).

  The projected video is an image of a (virtual) keyboard or touch panel, and the user performs an input operation on the image (corresponding to the image). When the input operation is performed, the object shape recognition apparatus 10 detects the input operation and uses it as input information (S11).

  Further, in the object shape recognition device 10, the shape following unit 17 follows the shape of the object 20 in the image by the camera 11 while the above image is projected on the display 18 (S <b> 12, shape following). Step). Information indicating the tracked shape is output to the display 18, and video projection processing or the like (S08 to S11) is performed based on the tracked shape. The above is the processing executed by the object shape recognition apparatus 10 according to the present embodiment.

  As described above, in the present embodiment, when the shape of the object 20 is recognized, a sound is generated when the user or the like hits the object 20. On the other hand, the object 20 is imaged at the timing when the sound is generated, and the position where the sound is generated in the captured image is detected. The shape of the object 20 is specified from the detected position. That is, in this embodiment, the shape of the object 20 is estimated by imaging the object 20, detecting the sound, and detecting the position where the sound is generated. These can be performed more easily than conventional shape recognition. As a result, according to the present embodiment, the shape of the object 20 can be easily recognized.

  If a sound is generated by hitting the object 20 as in the present embodiment, a sound can be easily generated from the object 20. By determining what the object is hit with (for example, the user's finger), information such as the finger is stored in advance, and the position where the sound is generated is reliably detected. By adopting such a configuration, the shape of the object 20 can be recognized more easily and reliably.

  In addition, since the portion struck by the user's finger as in this embodiment is recognized as a shape, when the shape is used as a range in which an image is projected by the display 18 as in this embodiment, the user can arbitrarily And a projection range can be designated easily. That is, in this embodiment, it is possible to recognize an appropriate shape. At this time, since it is not necessary to hold in advance information relating to the object 20 to be recognized and the background of the image to be captured, this embodiment is easy to implement in this respect as well.

  Furthermore, by storing in advance the information of the sound to be detected as in the present embodiment, it is possible to reliably detect the sound that is generated and to recognize the shape of the object 20 with certainty. However, it is not always necessary to store sound information. For example, a sound may be detected when a certain amount of sound (a sound having a volume exceeding a preset threshold) is generated. .

  Moreover, it is preferable to estimate a shape from a plurality of positions detected as in the present embodiment. According to this configuration, it is possible to appropriately recognize the shape of the object. In the present embodiment, the shape is estimated from three positions, but the shape may be estimated from more positions.

  Further, it is preferable to track the detected position as in this embodiment. According to this configuration, for example, even if the object 20 itself or the imaging direction of the camera 11 moves while detecting a plurality of positions, the shape of the object 20 can be estimated based on an appropriate position. That is, the shape of the object 20 can be recognized more appropriately. However, when the object 20 or the camera 11 is fixed or when a plurality of positions are detected from a single image in a short time, the above-described configuration is not necessarily required.

  Further, if the configuration as in the present embodiment is taken, a video is projected according to the object 20 that the user wants to project, and the above-described projection and superimposition can be appropriately performed. For example, an image is projected onto a notebook or notebook held by the user without a sense of incongruity. Further, if the projection is performed after following the shape as described above, the above-described projection and superimposition can be appropriately performed even if the imaging direction of the object 20 itself or the camera 11 moves. However, when the object 20 and the camera 11 are fixed as in the case of tracking the position, the above-described configuration is not necessarily required.

  Further, as described above, by recognizing the shape of the object 20 and projecting the image as a series of processes (including the follow-up process described above), the image can be projected in real time. The user's usability as a simple input device can be improved.

  In the above-described embodiment, the user generates a sound from the object 20 by hitting the object 20 or the like. However, it is not always necessary to generate a sound from the object 20. For example, when recognizing a user's input sound, when the surroundings are noisy and the sound of hitting the object 20 cannot be acquired, the sound of the user's voice may be detected by the microphone 12 and the sound detection unit 13. In that case, the position detection unit 14 detects a point pointed by the user's finger in the image captured by the camera 11 at the timing when the user's voice is detected. The detected point is a position corresponding to the shape of the object 20.

  In addition, when the surroundings are noisy, noise can be removed by analyzing noise in real time with the microphone 12 and creating a noise model.

  In the above-described embodiment, the positions detected by the user for detecting the shape of the object 20 are all detected by the user. However, a marker provided in advance on the object 20 may be used in combination. . That is, the position of the marker provided on the object 20 may be detected in the same manner as in the conventional method and used as a position used for shape estimation by the shape estimation unit 16. For example, one marker may be provided on the object 20 and the other two positions may be detected by the user hitting the object 20. As the marker, for example, a mark having a characteristic color or shape is used.

  In the present embodiment, the image obtained by being captured by the camera 11 is a normal image (captured by visible light), but is not necessarily the above-described image. What is necessary is just to be able to recognize the characteristics of the one hitting 20 (such as a user's finger). Specifically, for example, an image by infrared rays, a distance image, an image by thermography (temperature distribution) may be used.

It is a figure which shows typically the external appearance structure of the object shape recognition apparatus in embodiment of this invention. It is a figure which shows a mode that an object is hit by the user and the sound is generated when recognizing the shape of the object. It is a figure which shows the function structure of the object shape recognition apparatus in embodiment of this invention. It is a figure which shows the image | video projected in an object shape recognition apparatus. It is a figure which shows the hardware constitutions of the object shape recognition apparatus in embodiment of this invention. It is a flowchart which shows the process (object shape recognition method) performed with the object shape recognition apparatus in embodiment of this invention.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 10 ... Object shape recognition apparatus, 11 ... Camera, 12 ... Microphone, 13 ... Sound detection part, 14 ... Position detection part, 15 ... Position tracking part, 16 ... Shape estimation part, 17 ... Shape tracking part, 18 ... Display, 19 ... Image storage unit, 101 ... CPU, 102 ... RAM, 103 ... ROM, 104 ... auxiliary storage device, 20 ... object, 30 ... video.

Claims (9)

Imaging means for imaging an object that is a shape recognition target;
Sound detection means for detecting a predetermined sound;
Position detecting means for detecting a position corresponding to the shape of the object in an image captured by the imaging means at a timing when the sound is detected by the sound detecting means;
Shape estimation means for estimating the shape of the object from the position detected by the position detection means;
An object shape recognition system comprising: The sound detection means detects a sound generated from the object as the predetermined sound,
The position detecting means detects a position where the sound is generated as a position corresponding to a shape of the object;
The object shape recognition system according to claim 1. The sound generated from the object is the sound of the object being hit,
The position detecting means stores in advance information relating to the object that strikes the object, and detects the position where the object is struck based on the information as the position where the sound is generated.
The object shape recognition system according to claim 2.   The object according to any one of claims 1 to 3, wherein the sound detection unit stores in advance information related to a sound to be detected, and detects the predetermined sound based on the information. Shape recognition system. The position detecting means detects positions according to the shapes of the plurality of objects;
The shape estimation means estimates the shape of the object from a plurality of positions detected by the position detection means;
The object shape recognition system according to any one of claims 1 to 4. The imaging means images the object over a plurality of times,
A position follower for detecting a position corresponding to the position detected by the position detector in the time-changed image captured by the imager;
The shape estimating means estimates the shape of the object from the position detected by the position following means;
The object shape recognition system according to any one of claims 1 to 5.   The object shape recognition system according to any one of claims 1 to 6, further comprising projection means for projecting an image in accordance with the shape of the object estimated by the shape estimation means. The imaging means images the object over a plurality of times,
A shape follower for detecting a shape corresponding to the shape of the object estimated by the shape estimating unit in the time-varying image captured by the imaging unit;
The projecting means projects an image according to the shape detected by the position following means;
The object shape recognition system according to claim 7. An imaging step of imaging an object whose shape is to be recognized;
A sound detection step for detecting a predetermined sound;
A position detection step of detecting a position corresponding to the shape of the object in the image captured in the imaging step at a timing when the sound is detected in the sound detection step;
A shape estimation step for estimating the shape of the object from the position detected in the position detection step;
An object shape recognition method including:

Images