JP2007179338A - Information processor - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an information processor for surely achieving pointing, and for more improving entertainment property by arbitrarily moving this information processor without fixing it for detecting pointing. <P>SOLUTION: An information terminal device 10 supplies image data 22 picked up by an image pickup part 12 to a feature point extracting part 14 and a control part 16, and extracts the face included in the image data 22 and the feature points of the face by the feature point extraction part 14, and judges the movement of the face based on the position of the feature point by a feature judging part 18, and generates movement information 32, and generates a capture image as the face image to be displayed based on the image data 22 and the movement information 32 by the control part 16, and controls selection accompanied with pointing and information processing corresponding to this selection while being interlinked with an application, and displays the capture image to be supplied to a monitor 20. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to an information processing device, and more particularly to an information processing device such as an information terminal device, a mobile phone, or a game terminal device that operates an application according to an input image.

In general, there is a device that operates using an input image from a conventional camera. The operation principle of such an apparatus is that, for example, an input image is analyzed based on the difference between the background and the person and the operation of the person is operated based on the analysis result information. Patent Literature 1 is a line-of-sight / speech input device operated by an operator's line of sight and voice. Patent Document 2 is a portable device having a function capable of pointing. The movement direction and the movement amount are obtained by pattern matching, and the pointing function is given to the apparatus. Further, Patent Document 3 is a mobile phone with a pointing device function. Patent Document 3 uses image data obtained when the positions of the subject and the imaging unit are relatively changed, and sends a relative movement amount accompanying the change to the information processing apparatus for pointing.
Japanese Patent Laid-Open No. 10-39995 Japanese Patent Laid-Open No. 2003-288161 Japanese Patent Laid-Open No. 2004-312194

  However, in Patent Document 1, a camera that provides an input image is used in a fixed manner in order to realize such an operation in the apparatus. As shown by such usage, Patent Document 1 is not suitable for portable devices. This is because the image state acquired in the initial setting is maintained and the motion is analyzed with the above-described difference. It is difficult to use the mobile terminal device with the camera position fixed.

  In Patent Documents 2 and 3, a position is set in advance as a reference, and the mobile device is relatively moved to have a relative movement amount, which is reflected in the pointing amount. For this reason, the apparatus to which these are applied points by moving the apparatus in a desired direction.

  The present invention eliminates the drawbacks of the prior art, and provides an information processing apparatus that can be moved arbitrarily, and can be reliably pointed and improved in entertainment, without being fixed in the detection of pointing. Objective.

  In order to solve the above-described problems, the present invention provides an information processing apparatus that processes information in accordance with pointing. This apparatus captures an object scene and generates image data by imaging, and image data. Extraction means for extracting a feature point defining a face and a part of the face, a motion judgment means for judging the motion of the face based on the position of the feature point, and generating motion information, image data and motion information A control unit that generates a captured image as a face image to be displayed based on the screen and controls a selection associated with pointing and information processing according to the selection in conjunction with the application, and a display unit that displays the supplied captured image. It is characterized by including.

  According to the information processing apparatus of the present invention, the image data captured by the imaging unit is supplied to the extraction unit and the control unit, the face included in the image data and the feature points of the face are extracted by the extraction unit, and the motion determination unit To determine the movement of the face based on the position of the feature point, generate movement information, generate a captured image as a face image to be displayed based on the image data and movement information by the control means, and select and select with pointing By controlling the information processing according to the application and controlling it, and displaying the captured image supplied to the display means, along with the captured image, the facial feature points are also displayed to check the extraction status and Realize reliable pointing while moving, and improve entertainment by running applications Can can.

  Next, an embodiment of an information processing apparatus according to the present invention will be described in detail with reference to the accompanying drawings. Referring to FIG. 1, in the embodiment of the information terminal device 10 according to the present invention, the image data 22 captured by the imaging unit 12 is supplied to the feature point extraction unit 14 and the control unit 16, and the feature point extraction unit 14 supplies the image data 22. And a feature point of this face are extracted, a feature determination unit 18 determines the movement of the face based on the position of the feature point, generates motion information 32, and the control unit 16 generates image data 22 and motion information. By generating a captured image as a face image to be displayed based on 32, controlling the selection associated with pointing and information processing corresponding to this selection in conjunction with the application, and displaying the captured image supplied to the monitor 20 By displaying the facial feature points along with the captured image and confirming the extraction status, it is possible to realize reliable pointing while operating the application while moving arbitrarily You can improve the entertainment property.

  In this embodiment, the information processing apparatus of the present invention is applied to the information terminal apparatus 10. The illustration and description of parts not directly related to the present invention are omitted. In the following description, the signal is indicated by the reference number of the connecting line in which it appears.

  As shown in FIG. 1, the information terminal device 10 includes an imaging unit 12, a feature point extraction unit 14, a control unit 16, a feature determination unit 18, and a monitor 20. The imaging unit 12 has a camera function for capturing a scene and digitizing the acquired image. The imaging unit 12 outputs image data 22 including the photographed human face to the feature point extraction unit 14 and the control unit 16.

  The feature point extraction unit 14 has a function of extracting a human head from the image data 22 and further extracting the position of each part of the head as a feature point. This extraction process has the same function as the facial expression feature extraction unit described in Japanese Patent No. 3593057. Each part of the face is, for example, a face outline, eyes, mouth, nose, eyebrows, or the like. In other words, the extraction was analyzed using 68 FAP (Face Animation Parameters) described in the committee draft of Working Group 11 / Moving Picture Experts of Group (WG11 / MPEG) proposed on January 1, 1998 as feature points. To do. In order to realize this, the feature point extraction unit 14 includes an edge detection and feature point tracking function unit and the like (not shown).

  The edge detection function unit has a function of detecting the edge of the face image by scanning the supplied face image. The edge detection function unit extracts the contours of the eyes, mouth, nose, and eyebrows from the detected edges, and sends the extracted position data or coordinate data of each unit to the feature point tracking function unit. The feature point tracking function unit has a function of searching the supplied coordinate data, tracking the position of each obtained feature point, and detecting the coordinate data of each feature point. The feature point tracking function unit outputs the coordinate data 24 detected as the output of the feature point extraction unit 14 to the feature point determination unit 18.

  The feature point determination unit 18 includes a face frame calculation function unit 26, a face orientation determination function unit 28, and a mouth opening / closing determination function unit 30. The feature point determination unit 18 has a function of determining movement based on the supplied coordinate data 24 and generating movement information 32. The face frame calculation function unit 26 has a function of obtaining a user's face frame reflected in the camera from the coordinate data and calculating the moving direction and the moving amount of the head. The face orientation determination function unit 28 has a function of calculating the moving direction of the head from the coordinate data. The mouth opening / closing determination function unit 30 has a function of calculating opening / closing of the mouth from the coordinate data. The feature point determination unit 18 outputs the motion information 32 including the calculated face frame, face orientation, and opening / closing of the mouth to the control unit 16.

  The control unit 16 includes a pointer control function unit 34, an application control function unit 36, a display control function unit 38, and a captured image generation function unit 40. Based on the supplied image data 22 and motion information 32, the control unit 16 obtains information obtained by each of the pointer control function unit 34, the application control function unit 36, the display control function unit 38, and the captured image generation function unit 40. Are preferably supplied to each other and controlled in conjunction with each other. It is desirable that the application control function unit 36 controls application operations in conjunction with information supplied from other function units.

  The function of each functional unit is defined as follows. The pointer control function unit 34 has a function of controlling selection and movement by a pointer displayed according to the motion information 32 supplied. The application control function unit 36 has at least a function of how to control the application, and may also have whether or not to operate the pointer according to the motion information 32 supplied. The display control function unit 38 has a function of controlling the display of the face image on the monitor 20 based on the image data 22 and the motion information 32. More specifically, the display control function unit 38 has a function of updating a captured image, a facial feature point, and a face frame in real time to control display. The capture image generation function unit 40 has a function of extracting a face image from the supplied image data 22 and generating a capture image. The control unit 16 outputs display information 42 including the generated captured image data and cursor control information to the monitor 20.

  The monitor 20 has a function of displaying the supplied display information 42. A captured image is displayed on the monitor 20. This image becomes a mirror image. By displaying the captured image on the monitor 20, the user can check the detection state of the facial feature points.

  Next, feature points to be extracted and determination will be described. Each face part is given a reference number as shown in FIG. That is, left eye 44, eye head 46, eye corner 48, center of upper eyelid 50, center of lower eyelid 52, middle 54 between the corner of eye corner and upper eyelid, middle 56 between eye and center of upper eyelid, right eye 58, eye head 60 , 62 eye corner, 64 center of upper eyelid, 66 center of lower eyelid, 68 between center of eye and center of upper eyelid, 70 between eye corner and center of upper eyelid, tip of left eyebrow 72, middle of left eyebrow 74, right eyebrow Tip 76, middle right eyebrow 78, nose 80, left lip 82 as mouth, right lip 84, center of upper lip, ie man 86, lower lip center 88, middle 90 between man and left lip, man and right lip The position of the part is represented by an intermediate 92 between the center of the lower lip and an intermediate 94 between the center of the lower lip and the right end of the lip and an intermediate 96 between the center of the lower lip and the left end of the lip.

The face frame calculation function unit 26 calculates a reference width 98, a reference height 100, a face frame left end 102, a face frame right end 104, a face frame upper end 106, and a face frame lower end 108.
Reference width 98 = left eye 44-right eye 58,
Reference height 100 = lower lip center 88—left eye 44 or lower lip center 88—right eye 58,
Face frame left edge 102 = left eye 44-(reference width 98 x 1.0),
Face frame right edge 104 = right eye 58-(reference width 98 x 1.0),
Face frame upper end 106 = left eye 44− (reference height 100 × 0.8) or right eye 58− (reference height 100 × 0.8) and face frame lower end 108 are lower lip center 88+ (reference height 100 × 0.4).

The face orientation determination function unit 28 determines the right orientation and the left orientation based on the change in the right ratio and the left ratio. Hereinafter, the symbols x and y attached to the reference symbols represent x and y of the coordinate data. 2 and 3, the right ratio and the left ratio and the right ratio = ((right eye 58x−nose 80x) ÷ (face frame right edge 104x−face frame left edge 102x) × 100).
Left ratio = ((nose 80x-left eye 44x) ÷ (face frame right edge 104x-face frame left edge 102x) x 100)
Ask for. Further, the upward and downward directions are determined based on the change in the upward ratio and the downward ratio. 2 and 3, the upper ratio and the lower ratio upper ratio = ((nose 80y−right eye 64y) ÷ (face frame lower end 108y−face frame upper end 106y) × 100)
Lower ratio = ((lower lip center 88y−nose 80y) ÷ (face frame lower end 108y−face frame upper end 106y) × 100)
Ask for. Further, the orientation of the face may be determined based on a change in the center position (Cx, Cy).
Center position Cx = (Face frame left edge 102x + Face frame right edge 106y) / 2
Center position Cy = (Face frame upper edge 106y + Face frame lower edge 108y) ÷ 2
The mouth opening / closing determination function unit 30 has a function of determining an opening based on a change in the current and previous opening ratios. As shown in Fig. 4, the current and previous aperture ratios are
Current aperture ratio = (Current lower lip center 88y-Current human face 86y) ÷ (Current face frame lower end 108y-Current face frame upper end 106y) x 100), previous aperture ratio = (previous lower lip center 88y-previous 86y) / (previous face frame lower end 108y−previous face frame upper end 106y) × 100). The opening state is assumed to be larger than a preset opening threshold. When the current aperture ratio is not 0 and the previous aperture ratio is not 0, (current aperture ratio> previous aperture ratio) and (previous aperture state is closed) and ((current aperture ratio−previous aperture ratio)> In the case of (opening threshold), it is determined that the current opening state is an opening. In addition, when (previous opening ratio> current opening ratio), (previous opening state is opening state) and ((previous opening ratio−current opening ratio)> opening threshold), the current opening state is determined to be an opening.

  Next, the operation of the information terminal device 10 will be described with reference to FIG. The user activates the information terminal device 10. The user points the imaging unit 12 of the information terminal device 10 in front of the face. It is determined whether the face operation has been started (step S10). When the user starts the pointer operation by the movement of the face (YES), the process proceeds to the acquisition of the input image (to step S12). The pointer operation is performed by, for example, using a normal input interface of a key or a mouse, clicking a start button, pressing a key, or the like. If the pointer operation is not yet performed (NO), the process waits until the face operation is performed (to step S10). After this pointer operation, an input image is acquired by the imaging unit 12 (step S12). The acquired image data 22 is output to the feature point extraction unit 14.

  Next, the feature point extraction unit 14 analyzes the image, and extracts the face outline and feature points from the face image shown in the input image (step S14). The feature point extraction unit 14 outputs the extracted coordinate data 24 to the feature point determination unit 18.

  Next, the feature point determination unit 18 calculates a face frame (step S16). The feature point determination unit 18 outputs the motion information 32 including the face frame calculated by the face frame calculation function unit 26 to the control unit 16.

  Next, it is determined whether or not the face orientation has changed (step S18). The face direction determination function unit 28 of the feature point determination unit 18 calculates the face direction and detects the face direction. The method using the ratio described above is suitable for detecting the direction of the face that the user faces, and the method using the center position is suitable for detecting a change in the user's face position. The former ratio detection method may be determined by comparing a preset threshold value in each direction with each detected ratio. Further, “N% up” may be displayed by combining the ratio of the numerical value N% and the orientation of the face.

  When there is a change in the face direction (YES), the motion information 32 including the face direction calculated from the feature point determination unit 18 is output to the control unit 16, and the process proceeds to the pointer movement process (to step S20). If there is no change in the face orientation (NO), the procedure proceeds to a process for determining opening / closing of the mouth (to step S22).

  In the pointer movement process, the pointer is moved according to the motion information 32 supplied by the pointer control function unit 34, and the display information 42 including the captured image data and the control information of the moved cursor is monitored by the capture image generation function unit 40. Output to 20. The monitor 20 displays an image in which feature points and a face frame are superimposed on the input image. This display image is called a capture image.

  Next, it is determined whether or not there is opening / closing of the mouth (step S22). The mouth opening / closing determination function unit 30 of the feature determination unit 18 detects opening / closing of the mouth. In the case of opening (YES), the process proceeds to the application activation state confirmation process (to step S24). If the mouth is closed (NO), the process proceeds to a face operation end determination process (step S26).

  Next, in the application activation state confirmation process, the application control function unit 36 of the control unit 16 confirms whether or not the application is activated. If it is in the activated state (YES), the process proceeds to application termination processing (to step S28). If not in the activated state (NO), the process proceeds to the application activation process (to step S30).

  The application is terminated by the control by the application control function unit 36 (step S28). In addition, the application is activated under the control of the application control function unit 36 currently pointed to by the pointer (step S30). After these processes, the process proceeds to a face operation end determination process (to step S26).

  Next, it is determined whether or not the face operation is finished (step S26). If the face operation has not been completed (NO), the process returns to the acquisition of the input image (to step S12). If the face operation is finished (YES), the process proceeds to the end, and the face operation process is finished.

  In this manner, the facial feature points are extracted from the input image, the face frame, the face orientation, and the opening / closing of the mouth are determined, and the device pointer can be operated by controlling the operation of the device based on the determined results. In addition, the information terminal device 10 is more effective in a portable device because the portion other than the face may be unfixed.

  The information terminal device 10 can be applied to a selection operation of a list menu, a scroll operation, and the like in addition to an example shown by movement of a pointer and an application.

  A more specific example will be described with a game. The control unit 16 includes a game control function unit as the application control function unit 36. Further, the control unit 16 has a function of superimposing a game object on the capture image generation function unit 40, and has a function of generating an image obtained by synthesizing the capture image and the game object. The captured image 110 shown in FIG. 6 is displayed on the monitor 20 using these functions.

  Next, the operation of the information terminal device 10 in this case will be described. In the information terminal device 10, the game application has already been activated by the application control function unit 36. The game application of the present embodiment operates according to the procedure shown in FIG. An object is added as the game application starts (step S30). The captured image is superposed and synthesized with the objects 112, 114, and 116 on the face image by the captured image generation function unit 40. As shown in FIG. 6, the objects 112, 114, and 116 cause, for example, rice balls, corn, fish, and the like to descend from the upper part of the screen of the monitor 20 toward a predetermined position. Objects should be generated at an appropriate frequency.

  Next, it is determined whether the mouth has been opened or closed (step S32). For this determination, the opening / closing of the mouth is checked from the movement information 32 including opening / closing of the mouth obtained by the feature point determination unit 18. If the mouth has been opened or closed (YES), the process proceeds to the position confirmation process (to step S34). If the mouth has not been opened or closed (NO), the process proceeds to the object moving process (go to step S36).

  Next, in the position confirmation process, it is determined whether or not the position of the object matches the position of the mouth (step S34). The position of the mouth feature point is compared with the position of the object. The position of the mouth feature point is the feature point in the mouth of the face shown in FIG. In the captured image 110 of FIG. 6, there are six symbols ◯ arranged around the mouth. Both the positions coincide with each other when, for example, they overlap within a certain range such that the center position of the object is included in the range surrounded by the mouth feature points. If both positions match (YES), it is determined that the object has been picked up by the mouth, and the process proceeds to object processing (to step S40). If the two positions do not match (NO), the process proceeds to the object movement process (to step S36).

  Next, in the object processing, the given object is deleted, and a predetermined game score is added (step S38). Next, the process proceeds to a game clear confirmation process (to step S40).

  In the game clear confirmation process, it is determined whether or not a predetermined clear condition has been achieved (step S40). The clear condition is, for example, erasing a certain number of objects or obtaining a certain score. If the condition is satisfied (YES), it is determined that the game is over, the process proceeds to end, and the game application is ended. If the condition is not satisfied (NO), the game is continued and the process proceeds to the object movement process (to step S36).

  Next, in the object moving process, the remaining object is moved (step S36). Objects that have reached the bottom of the screen disappear from the screen due to object movement. Next, it is determined whether or not the object has reached the outside of the display range with this movement (step S42). If the object is at the lowermost end (YES), the process proceeds to the object deletion process to be deleted from the screen (to step S44). If the object is not at the lowermost end (NO), the process returns to the mouth open / close check (to step S32).

  Next, the object is deleted (step S44). The corresponding object is lost without being scored. The game application counts the total number of lost objects or total points.

  Next, it is determined whether or not the total number of lost objects counted or the total score has reached a preset number or score (step S46). If the total number of lost objects or the total number of points lost has not reached the preset number or points (NO), the process returns to the mouth open / close check (to step S32). If the total number of lost objects or the total goal scored reaches a preset number or score (YES), the game is over when the game is over.

  By superimposing the game operation and the game object on the captured image in this way, it is possible to make an expression as if the user exists in the game and the game progresses by the user's action. There is an effect of further improving entertainment properties.

  The information terminal device 10 is not limited to the above-described game. For example, in the game in which an object indicating a user is arranged at the bottom of the screen, and an object indicating an enemy fires a missile-like bullet from the top of the screen. It can be used for the movement operation of the object indicating the user with the face direction, and the opening and closing of the mouth can be used for the missile launch operation from the user.

  Compared with Patent Document 3, the information terminal device 10 according to the present embodiment uses face detection, and therefore can detect the absolute position of the face on the screen in addition to the temporal relative change. In Patent Document 3, the position is detected only by the amount of movement from the set reference position. Since the information terminal device 10 can determine the position of the pointer based on the position of the face rather than the movement, the background does not have to stop in obtaining the initial position. Since the information terminal device 10 detects the facial expression, the pointer can be operated only with the face portion without taking a large action or action by the user.

  Further, since the information terminal device 10 can reflect the movement of the user in the action of the game, the information terminal device 10 can have a sense of presence and empathy as if it were projected on the virtual game world. As a result, the information terminal device 10 can execute a game with higher entertainment characteristics as compared to Patent Document 3.

It is a block diagram which shows the schematic structure of the information terminal device to which the information processing apparatus of this invention is applied. It is a figure which shows the position of the face extracted as a feature point of a face in the feature point extraction part of FIG. It is a figure explaining the positional relationship of the feature point used for determination of the face direction using the feature point of FIG. It is a figure explaining the positional relationship of the feature point used for judgment of opening of the mouth using the feature point of FIG. It is a flowchart explaining the operation | movement procedure in the information terminal device of FIG. It is a figure which shows the monitor image which synthesize | combined the character image and object which are displayed on the monitor of FIG. It is a flowchart explaining the operation | movement procedure in the game application at the time of applying to the information terminal device of FIG.

Explanation of symbols

10 Information terminal equipment
12 Imaging unit
14 Feature point extraction unit
16 Control unit
18 Feature judgment unit
20 Monitor

Claims (4)

In an information processing apparatus that processes information according to pointing, the apparatus includes:
Imaging means for imaging the object scene and generating image data by imaging;
Extracting means for extracting a feature point defining a face and a part of the face included in the image data;
A movement determining means for determining a movement of the face based on the position of the feature point and generating movement information;
Control means for generating a captured image as a face image to be displayed based on the image data and motion information, and controlling the selection associated with the pointing and information processing corresponding to the selection in conjunction with an application;
An information processing apparatus comprising: display means for displaying a supplied captured image. The apparatus according to claim 1, wherein the movement determination unit calculates a face frame from the feature points;
An orientation determination function block for determining the orientation of the face from the feature points;
An information processing apparatus comprising: an opening / closing determination function block that determines opening / closing of the mouth from the feature point. 3. The apparatus according to claim 1, wherein the control means controls a pointer indicating a pointer indicating the position of the pointing according to the motion information;
A display control function block for controlling display of a face image on the display means based on the image data and motion information;
An information processing apparatus comprising: an image generation function block that generates the capture image.   4. The information processing apparatus according to claim 3, wherein the control means includes a functional block that instructs at least how to control the application.

Images