JP2001216069A - Operation inputting device and direction detecting method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an operation inputting device and method for allowing a driver to safely perform an operation without gazing on-vehicle equipment or the like. SOLUTION: Two kinds of gestures, that is, a shape gesture and a direction gesture are detected, and an operation mode is selected by one gesture, and a parameter in the operation mode is changed by the other gesture.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an operation input device and a method for inputting operations to various information devices such as on-vehicle information devices.

[0002]

2. Description of the Related Art In recent years, many information devices and electric devices such as an audio device, a TV, a video device, a mobile phone, an air conditioner, and the like have been mounted in automobiles. In addition to making phone calls in the car, you can read and write e-mail and even access the Internet. This trend will continue to grow,
With the introduction of automatic toll collection systems and safe driving support systems, the car is about to become a running computer.

In addition, it is difficult to prepare all operation panels for these various devices, and the operation units are being shared. At present, there is generally a situation in which there are operation buttons for car audio, operation buttons for air conditioner, operation buttons for car navigation or a remote control, and operation is performed using each of them.

[0004] However, in some in-vehicle devices, the status of all devices is displayed on one display, and operations can be performed by a common input device. For example, the display has a touch panel function, and can be operated by touching a menu displayed on the screen. Since different screens can be displayed on the display depending on the situation, all the devices can be operated. For example, display "air conditioner", "audio", "car navigation"
If you touch "Audio", an audio operation panel will be displayed, and if you touch the "Volume Up" button in it, the volume will increase.

Further, there are many products that operate the displayed menu screen using a device such as a remote controller instead of a touch panel. In such a case, typically, a remote controller is provided with a button for instructing four directions, and there is a type in which a menu is selected by operating the button.

[0006] As the types of in-vehicle devices increase in this way, the types of operations performed by the driver have increased dramatically. In particular, with the advent of car navigation, it can be said that the types of operations have greatly increased. For this reason, the case where the driver performs an operation during driving or for a short time such as waiting for a traffic light is increasing.

Conventionally, audio operations (volume adjustment,
Music selection) and air-conditioning operation (temperature adjustment, air volume adjustment), but in recent years, switching the display screen of car navigation, scrolling the map to find facilities near the current location, Sometimes used.

[0008] By increasing the number of such operations, the driver's attention during driving is disturbed, which is likely to cause an accident or the like. In fact, the use of mobile phones while driving increased the number of accidents, and the revision of the Road Traffic Law banned the use of mobile phones while driving. However, when using a hands-free device that allows a user to talk without holding a telephone in his hand, use while driving is permitted.

As seen in this example, it is important for safety to be able to do other things (in this case, telephone) without moving the driver's gaze.

On the other hand, the operation of the on-vehicle equipment will be described.
Usually, the driver's attention is deviated from the front because the operation is performed while looking at the operation panel. An increase in such operations indicates an increase in danger during driving.

Further, as in the above-mentioned one example, if the operation of the devices is centralized on the display and the common input device, for example, conventionally, to raise the temperature of the air conditioner, the temperature knob is turned. What should have been to press the temperature rise button, select "Air Conditioner" from the main menu,
After the operation panel of the air conditioner is displayed,
Pressing the button will increase the movement of the driver's gaze,
The danger increases.

In order to solve these problems, it is conceivable to use a hand gesture, and several devices have been proposed. Japanese Patent Application Nos. 9-199647 and 8-3061
In 02, using a hand movement detection device using an optical sensor,
A device for inputting directions by hand and performing operations has been proposed. However, normally, when a person inputs hand movements, for example, when the hand repeatedly performs “right” and “right”, the hand returns to “left” during that time. According to this proposal, when such a movement is made, "right", "left", and "right" are input.

[0013]

SUMMARY OF THE INVENTION An object of the present invention is to solve the current situation in which it is difficult for a driver of an automobile to operate a vehicle-mounted device stably while gazing forward. It is another object of the present invention to enable a shape gesture and a direction gesture, which are effective at that time, to be input more efficiently and stably.

In addition, the present invention assumes that not only the driver but also the passenger is operating, and invents an effective operation method when operating while watching the screen. It is also an object to operate more safely and stably in a special environment such as a vehicle. It is another object of the present invention to establish a general-purpose gesture input method that can be used not only in a vehicle but also in various devices.

[0015]

According to the present invention, a first detecting means for detecting a shape and a movement of an object to be detected is different from a shape and a movement detected by the first detecting means. Second detection means for detecting the shape or movement of the object, operation selection means for selecting a specific operation based on the detection result of the first detection means, and selection by the operation selection means A parameter change unit that changes a parameter that can be changed in the performed operation based on a detection result of the second detection unit.

Further, based on a search result of the shape detecting means for detecting a shape of the object to be detected, a direction detecting means for detecting a direction of movement of the object, and a search result of the shape detecting means.
An operation selecting unit for selecting a specific operation, and a parameter changing unit for changing a changeable parameter in the operation selected by the operation selecting unit based on a detection result of the direction detecting unit. I do.

Further, the operation selection means associates a plurality of operations with the shape of the object, and when the shape detection means continuously detects the same shape gesture, the operation selection means is associated with the shape. A plurality of operations are sequentially switched and selected.

Further, the shape detecting means detects the shape of the object when detecting that the movement of the object is stationary or small for a predetermined time or more.

Further, the apparatus has display means for displaying information on selectable operation or changeable parameters. The display means includes an image showing a selectable operation mode and a shape gesture for selecting the operation mode, Alternatively, it is characterized in that a changeable parameter and an image indicating a direction for changing the parameter are displayed together.

[0020] The present invention is characterized in that it has a voice presenting means for presenting information on selectable operation or changeable parameters by voice.

[0021] Further, the present invention is characterized in that voice presentation is performed when the processing of the shape detecting means or the direction detecting means is delayed.

Further, there is provided a sound presenting means for presenting to the operator with sound when the detection by the shape detecting means, the detection end, the detection failure, and the direction detection by the direction detecting means occur. .

Also, the sound presenting means presents different sounds depending on the type of operation and the type of detected direction.

Also, a method of analyzing a trajectory of position information of an object to be detected and detecting a direction of the object,
When the object moves by a predetermined distance in a predetermined direction and moves by a predetermined distance in a direction opposite to the direction, the movement is detected as a movement in the direction.

A method for detecting the direction of an object by analyzing a trajectory of position information of the object to be detected,
After the object remains in a predetermined range in a state of rest or little motion for a predetermined time, moves in a predetermined direction by a predetermined distance, moves by a predetermined distance in a direction opposite to the direction, and again performs a predetermined operation. It is characterized in that when the robot stays still or has little motion for a time, the motion is detected as the motion in the direction.

Further, the direction detecting means detects the movement in the direction when the object moves by a predetermined distance in a predetermined direction and moves by a predetermined distance in a direction opposite to the direction. It is characterized by.

Further, the direction detecting means moves the object in a predetermined direction for a predetermined distance after the object stays in a predetermined range in a stationary state or a state of little movement for a predetermined time, and When the user moves in the direction by a predetermined distance and stays still for a predetermined time or stays in a state of little movement, the movement is detected as the movement in the direction.

The sound presenting means presents a first sound when moving in a predetermined direction, moves back to the opposite side of the direction, and returns a second sound when movement in the direction is detected.
The present invention is characterized in that the sound of

Further, the sound presenting means is characterized in that the first sound presents a different sound depending on the moving direction, and the second sound presents the same sound regardless of the direction.

Further, the direction detecting means is characterized by distinguishing between an intentionally moving operation and an unintentionally occurring operation.

Further, the direction detecting means includes means for detecting which one of a plurality of predetermined directions has moved, and means for removing a return operation occurring between successive movements in the same direction. It is characterized by having.

Further, the direction detecting means has means for detecting a distance of the moved object from the sensor, and a distance in a direction opposite to the direction is long during a continuous movement in a predetermined direction. When a movement is detected, it is determined that the movement is a returning movement and is removed.

Further, the direction detecting means has means for measuring a time interval between movements of the object, and a movement in a direction opposite to the direction is detected during a continuous movement in a predetermined direction. When the time interval is equal to or shorter than a predetermined length, the movement during this time is determined to be a returning movement and is removed.

[0034] Further, the present invention is characterized by having a sound presenting means for presenting different sounds in the detected direction.

Further, the direction detecting means detects a trajectory in which the starting point and the ending point of the movement of the object are close to each other.

Further, a shape detecting means for detecting the shape of the object to be detected, a menu display means for displaying an operation menu associated with the shape detected by the shape detecting means, And a menu selection unit for selecting the operation menu according to the shape detected by the detection unit.

Further, the menu selection means presents different shapes when the operation menu is hierarchically traversed.

A direction detecting means for detecting the direction of the movement of the object to be detected; a menu displaying means for displaying an operation menu associated with the movement direction detected by the direction detecting means; When the direction of movement is continuously detected by the direction detecting means, 1
Menu selection means for selecting an operation menu different from the operation menu detected only once.

Further, the shape detecting means for detecting the shape of the object to be detected, the direction detecting means for detecting the direction of the movement of the object to be detected, and the detected shape and the moving direction are associated with each other. Menu display means for displaying the operation menu, and menu selection means for selecting the operation menu in association with the detected shape and movement direction.

[0040]

Embodiments of the present invention will be described below with reference to the drawings.

<Overview> FIG. 1 is an example of an embodiment of the present invention. The first gesture detection unit 1 and the second gesture detection unit 2 detect different types of gestures. Then, the operation type selection unit 3 selects an operation type according to the detection result of the first gesture detection unit 1.
The parameter change unit 4 changes a changeable parameter in the operation selected by the operation type selection unit 3 according to the result of the second gesture detection unit 2.

The operation input processing unit 5 determines what operation has been performed according to the results of the operation type selection unit 3 and the parameter change unit 5. The operation execution unit 6 executes the determined operation. The operation input processing unit 5 includes a first and second gesture detection unit and operation type selection unit 3, a parameter change unit 4.
, So that they do not work. Therefore, only one of the gestures can be detected depending on the situation.

<Detailed Configuration> FIG. 2 shows FIG. 1 more specifically. Here, the first gesture detection unit is a shape gesture detection unit 7 that detects a shape gesture, and the second gesture detection unit is a direction gesture detection unit 8 that detects a direction gesture.

Here, the "shape gesture" is a "gesture expressed in the form of a hand", and the "directional gesture" is a "gesture indicating the direction in which the object has moved". In this,
In the case of a shape gesture, the hand does not necessarily have to be moving, and the word “gesture” may not be easily applied in that sense.

<Detailed Description of Detecting Means> Here, the two gesture detecting units will be described in detail. As a specific example, the above-mentioned shape gesture is a hand shape such as “goo”, “choki”, and “par” that is presented as a rock-paper-scissor. Alternatively, it refers to various shapes expressed by hands, such as "Which of the five fingers should be extended" or "Make a circle with the index finger and thumb".

FIG. 3 shows an example of the shape gesture. Originally, a “gesture” includes a meaning involving movement, and a still hand shape may not always be called a gesture.However, when presenting these shapes, the operator must use other shapes from other shapes. Since the shape of the hand is transformed into the shape of, it is treated as a gesture in a broad sense in that sense. Also,
Considering the detection processing method, it is relatively easy to treat them as stationary shapes, but in practice, there may be a combination of these shapes in a time series, or a shape accompanied by a movement.

Specifically, as a means for capturing the shape of the hand, it is possible to detect the shape by imaging the hand using a small CCD camera or the like and analyzing the image. However, especially when applied to in-vehicle equipment, the problem is that the detection accuracy decreases because the inside of the car is bright during the daytime and the nighttime is dark, and the external light condition fluctuates greatly, and the image obtained by the camera fluctuates greatly. There is. Each time a person or object crosses the front of the camera, there is a risk of erroneous recognition.

The present inventors have developed an image input device capable of detecting an object shape at high speed by a light emitting means and an image sensor operating in synchronization with the light emitting means, as described in Japanese Patent Application Laid-Open No. 10-177449. This is an apparatus that typically irradiates an object with infrared light, and extracts only the object reflected light as an image using an image sensor that operates in synchronization with the infrared light. Since the intensity of the reflected light is generally inversely proportional to the square of the distance to the object, a nearby object can be easily cut out from a distant background.

When this input device is used, since the device itself has a light emitting means, detection can be performed even under conditions where it is difficult to capture an image with a normal CCD camera, such as at night. Further, since the light emitting means uses near-infrared light which is invisible to human eyes, it does not hinder the driver. In addition, since the reflected light does not reach the background object at a long distance and the reflected light does not reach the object, malfunction due to unrelated movement or the like can be reduced.

A method of recognizing the shape of a hand obtained from these image input devices and determining whether or not the hand matches a predetermined shape gesture will be described. For example, when detecting a difference in shape due to the number of fingers extended, it is possible to detect a difference in shape by setting an area where a stretched finger will cross and counting the number of fingers crossed.

As another method, considering the outline of the shape of the hand, the fingertips protrude in a mountain shape, and the dents between the fingers are depressed in a valley shape. You can know the number of fingers. When the fingertip is facing the camera, the fingertip is a point closer to the camera than the surroundings, and the amount of reflected light is maximized. By counting the number of the maximum points, the number of fingers facing the camera can be detected.

In order to recognize a free hand shape,
A technique called pattern matching can be used. In this method, the shape to be recognized is registered in advance as a pattern, the degree of difference from the input shape is calculated, and if there is a similar shape, it is determined that the shape has been input. As a method of calculating the degree of difference, there is a method in which patterns are simply overlapped, the degree of overlap is calculated, or a method is used in which a pattern is mapped to a feature space and the distance is used to make a judgment.

<Start Shape Recognition after Stilling> In the case of shape recognition, the more accurate the process, the longer the processing time. Therefore, if the recognition process is performed for each frame of the input image, the processing cost increases. When presenting the shape, it is desirable to start the recognition process when the operator presents the shape and stands still for a certain period of time. This is because when the operator intentionally produces a shape, the operation of stopping the operation is natural.

Further, since the recognition process only needs to be performed when a stationary state is found, the calculation cost can be reduced. Also, a motion that has been moved without intention to present a gesture will not be erroneously recognized as a gesture presentation unless it is stationary.

On the other hand, the direction gesture can be performed by extracting the hand movement direction from an image obtained from such an image input device. For example, Japanese Patent Application Laid-Open No. Hei 10-1774
When the image input device of 49 is used, the background is not shown in the image, and only the shape of the hand is shown. Therefore, if the center of gravity of this image is calculated, the position of the hand can be obtained. By tracking the trajectory, a basic movement can be detected.

When the fingertip is moved almost without moving around the wrist, if the center of gravity of the entire hand is calculated, the accuracy of the detected movement may be poor. In such a case, it is conceivable to perform processing in which the shape of the hand is carefully considered.

For example, when the finger is extended toward the camera, the tip of the finger is closest to the camera, and therefore the reflected light is the largest, and is often the maximum point in the image. Detect such points and find their closest ones,
Alternatively, position information can be obtained as an evaluation value that is more sensitive to an actual movement than by obtaining the center of gravity of the entire hand by calculating the average of the maximum points and the like. in this case,
Finding the correspondence between image frames of a plurality of local maximum points, etc., prevents the position behavior when the local maximum points that existed at the edge of the image goes out of the image, etc. be able to.

In any case, it is preferable to detect positional information from the obtained hand shape data and analyze the trajectory to detect a motion gesture. Normal CC
Even when a D camera is used, if the shape of the hand can be properly cut out based on color information or the like, it is possible to extract position information therefrom. A method for detecting the direction gesture will be described in detail later in the embodiment.

<Operation State Transition> FIG. 4 shows a state transition diagram in an operation configured using these two types of gesture detection. At first, nothing is done.
Here, detection is performed by the first gesture detection unit. When any gesture is presented, it is detected and enters one of the operating modes. For example, when gesture 1 is presented, a mode for operating a car navigation view is entered,
When the gesture 2 is presented, the operation mode of the air conditioner is entered, and when the gesture 3 is presented, the operation mode of the audio operation is entered. Within each operation mode, several parameters can be adjusted by four direction gestures.

For example, in the audio operation mode,
Presenting an upward gesture increases the volume, and presenting a downward gesture decreases the volume. In addition, the next tune such as a music CD can be selected by presenting the gesture in the right direction, and the previous tune can be selected by presenting the gesture in the left direction. Even if only audio is considered, there are various operation targets such as a music CD, a music MD, a cassette tape, a radio, and a TV in some cases. First, when the gesture 3 is presented, the operation mode is changed to the audio operation mode. At that time, the operation of the currently moving target can be performed.

For example, if a music CD is playing, music selection and volume adjustment of the CD can be performed. The operation target can be switched by repeatedly presenting the gesture 3. For example, if it is repeated twice, the music MD is switched, and if it is presented again, the music MD is switched. At this time, the change parameter by the direction gesture may change depending on the operation target.

For example, in the case of a CD or MD, the next song and the previous song are selected by right and left direction gestures. In the case of a radio, the next station and the previous station are selected in the order in which the radio stations are registered. Become. Since the volume adjustment is frequently used for any operation target, the volume adjustment is performed for the gesture in the up and down directions in any case.

By remembering the correspondence between the shape gesture and the operation target, the driver can select an operation mode without looking at the operation panel. In addition, since the operation in the selected operation mode is also very easy to remember, such as the volume of the volume with the up and down direction gesture, all the series of operations can be performed while keeping the line of sight ahead. Can be done without looking.

For example, when a car navigation is used, a map is often displayed on the screen. At this time, if an audio operation is to be performed, a gesture of the display is presented and the audio operation panel is immediately displayed. Is displayed. Normally, the main menu is displayed once, an audio operation item is searched from the main menu, and the audio operation panel is displayed only after selecting the item.

As compared in this way, the effect of quickly accessing frequently used functions by the gesture presentation is significant.

<Gesture Icon> The operation using these gestures is very simple, and frequently used shape gestures can be easily stored. In addition, as for the direction gesture, the movement and the effect (such as raising or lowering the volume) are well associated with each other, so that it is easy to remember. However, in some cases, such as when a user has just started using or after not using it for a long time, he or she may forget what kind of gesture and what could be controlled. In such a case, it is very inconvenient if the user must take out and read the instruction manual.

Here, by further displaying the icon displaying the gesture on the screen at the same time, even if the operation is forgotten, the gesture to be performed can be easily known. This is shown in FIG. In this figure, the icon 9 indicating the shape of the gesture and the operation that can be performed when the gesture is presented are displayed together at the bottom of the screen. in this way,
By displaying the shape gestures that can be executed and what can be operated with them, if you forget them, you can immediately recall by looking at this screen.

Looking at the gesture icon on this screen for each operation is the same as looking at the operation panel.
The driver's attention will be distracted, but in this case, the safety is maintained because the driver only looks when he forgets the operation. Rather, it is more dangerous to forget the operation and look at the instruction manual.

<Voice Feedback> If an operation is performed without looking at the screen, sometimes it is difficult to know what operation mode is currently in operation and what can be operated now. This is different from forgetting a gesture, as described above, and can be caused by distraction. For example, when a gesture for audio operation is presented to switch from a radio to a CD, suddenly hitting a favorite song recently or suddenly speaking during operation, the user may be distracted. Easy to happen by.

In order to cope with such a situation, the present invention adds a voice feedback function for notifying the user of the current situation using voice according to the situation. For example, in the above-described example, after presenting the audio operation gesture, when something is distracted and the user does not understand, the audio feedback “audio operation” is given. Alternatively, the audio feedback may be changed according to the situation, such as "Radio operation" if the radio is currently on, and "CD operation" if the CD is on. Voice feedback may be provided each time an operation is performed.

However, when the user gets used to the operation, the audio operation gesture is followed by the direction gesture (such as increasing the volume).
Are often presented in succession, and each time voice feedback occurs, it becomes annoying. Therefore, when there is no next operation for a while after the audio operation gesture is presented, it is desirable to judge that the user has forgotten the situation and return audio feedback. Similarly, when a certain period of time has elapsed after returning the voice feedback, what can be operated by the direction gesture may be returned by the voice feedback. For example, the voice is fed back, such as "Adjust the volume up and down, select the channel with left and right".

FIG. 6 is a block diagram showing this mechanism. The operation input processing unit 10 starts the timer 12 when the shape gesture is input. The timer 12 informs the operation input processing unit 10 when a certain time has elapsed, and in accordance with the notification, the operation input processing unit 10 instructs the audio feedback unit 13 to perform audio feedback.

FIG. 7 is a flowchart showing this process. In the initial state, only the shape gesture is detected (S14), and when the shape gesture is detected (S15), the operation mode is changed (S1).
6) Here, a timer is started (S17). Subsequently, direction gesture detection (S18) is started. In this loop, it is determined whether a predetermined time has elapsed since the timer was started (S20), and when it is determined that the timer has elapsed, voice feedback is generated (S20). S19).

In this flowchart, the path for returning to the initial state for detecting the shape gesture is omitted. However, this is configured so that the path returns after a certain period of time or when a specific gesture is performed. it can.

<Sound Feedback> It has been described that it is effective to feed back the current state by voice when the user is lost in operation. effective. With a conventional operation panel using buttons, the operator can know that the button has been touched, and if the button is pressed, it can be used as a clue to determine whether the button has been pressed properly, and the volume changes. By actually feeling the effect such as a change in the music, it is possible to know that the operation has actually been performed.

However, when performing an operation by gesture, there is a problem that an operation feeling is poor because the user does not touch anywhere. The same applies, for example, to operations on a computer.

For example, when a certain application is started by double-clicking a desktop icon with a mouse, it usually takes time to load a program from a storage device such as a hard disk and display it on a screen. After that, it does not start instantly. Therefore, after double-clicking, it is uneasy to know whether or not the operation was successful until the startup screen is truly displayed.

In many computers and software, in order to deal with this problem, when the program is started by double-clicking, the display of the mouse cursor is changed immediately to notify that the start is being performed. I have. For example, the arrow-shaped cursor may change to an hourglass to indicate that it is preparing for activation. This fact indicates that some kind of feedback is important.

On the other hand, considering that the driver operates the in-vehicle device during driving, it is not appropriate to provide feedback based on changes on the screen, such as changing the shape of the cursor. This is because you have to divert your attention from the front to see the feedback. Returning the feedback by sound can be confirmed by the driver without distracting the driver from the front, and is extremely effective particularly for a gesture-based operation input device as in the present invention. The feedback by voice takes a long time, so it feels annoying every time it is heard. However, since the feedback by sound is instantaneous, there is no hassle, and it is possible to confirm that the operation has been completed, and there is an effect that the user can feel relieved.

Further, if the sound is changed according to the type of the gesture, it can be determined whether the correct gesture has been performed. For example, when the gesture of "view" is presented, the sound of "pip" is fed back, when the gesture of "air conditioner" is presented, the sound of "pop" is presented, and when the gesture of "audio" is presented, the feedback sound is "pop". Then, even if the user intends to select the “audio” operation and hear a “beep” sound, it is possible to know that the operation was not performed correctly.

In practice, it may be rare that the user clearly remembers that the sound “audio” and the sound “pop” are related, but the gesture of “audio” always makes the sound “pop”. When listening to the sound, sometimes when a different sound is heard, it feels strange. Therefore, it is possible to know "it did not work correctly" without knowing exactly what was wrong.

Also, it has been described that when performing shape recognition, it is desirable to perform shape recognition processing only when a stationary state is detected. When such processing is performed, a sound is emitted when a stationary shape is found, and a sound is emitted again according to the gesture when a gesture is detected. For example, when a stationary state is detected and shape recognition is started, a beep is sounded, and when shape detection is successful, the above-mentioned "beep" and "beep"
A beep sounds, and if the detection fails, a beep sounds. This makes it possible to know that the recognition has failed, and to know that the gesture has been recognized unintentionally even when the gesture has been recognized.

Whether the sound should be produced in two stages depends on how long the recognition process takes. For example, if the recognition starts when the stationary state for 0.2 seconds is met and the recognition processing ends in 0.1 seconds, the gesture is detected 0.3 seconds after the operator presents the shape. In this case, two-stage sound presentation may not be necessary. Conversely, if the recognition is started when the stationary state is achieved for 0.2 seconds and the recognition process is completed in 1 second, it is not known whether the gesture is detected until 1.2 seconds have passed since the operator presented the shape. In this case, two-stage sound presentation is effective.

The expressions such as "beep", "beep", and "blow" described in this example merely indicate that the type of sound is different, and there is no limitation as long as the sound is short-time. However, 3
It is desirable that the sound produced by one shape gesture be different to the extent that the difference can be easily recognized to a certain extent. For example, the sound when recognition fails is relatively tied to "failure", such as "boo". It is desirable to use easy sounds.

<Other Gestures> In the present embodiment, an example has been described in which the first gesture detector detects a shape gesture, and the second gesture detector detects a direction gesture. However, the embodiment is not limited to this, and two groups of gestures belonging to different categories may be selected from all types of gestures, and the operation mode selection and the parameter adjustment may be performed using the selected groups.

As a special example, the case where the first and second gesture detection units detect the same type of gesture is also included. For example, there is a method of first selecting four operation modes with a direction gesture and then changing parameters with a direction gesture.

In this case, when a gesture in the same direction is detected, the determination of which gesture is to be controlled depends on whether the operation mode is currently selected. If the operation mode is not selected, the gesture is determined to be a gesture for selecting an operation. If the operation mode is already selected, the gesture is determined to be a gesture for changing a parameter.

In the above example, the direction gesture is
It was only determined whether there was a gesture in either direction. Instead, the value can be changed continuously by moving the hand up, down, left, and right. For example, in the above-described embodiment, the volume gradually increases each time the upward gesture is performed, and when it is desired to greatly increase the volume, the upward gesture must be presented many times. Move this up and down to change the volume accordingly. Then, when the camera is stopped for a certain period of time or when the movement in the left-right direction is detected, the sound volume is fixed. This method is also suitable for those who want to change parameters quickly.

Although only the vertical and horizontal directions have been described, the movement in the depth direction can also be used. JP-A-10-1
By using the image input device invented in 77449, the intensity of the reflected light depending on the object distance can be obtained as an image.
Since the position in the depth direction can be obtained stably,
It is suitable for such usage.

<Rotation Gesture> As another example, a rotation gesture can be used instead of a direction gesture. That is, by presenting the shape of a hand having a knob and performing a hand motion such as turning the knob, the amount of rotation of the movement is detected, and this is set as a parameter. The feeling of turning the tone adjustment knob can be realized.

<Input Method of Direction Gesture> The input method of the direction gesture, which is one element of the above-described embodiment, will be described in detail. There is an important point in realizing the input of the direction gesture. That is, gestures in the same direction can be input repeatedly.
It is important that the detection be stable at that time. For example, in a hand gesture input device proposed in Japanese Patent Application No. 9-199647, an input is performed by a hand crossing the front of the device.

However, if such a device repeats a gesture in the same direction, a malfunction may occur. That is, for example, when repeatedly inputting “right” and “right”, the hand must once return to the left between two rightward gestures. At this time, this is erroneously recognized as a left gesture, and the input of “right” and “right” is input as “right”, “left” and “right”. To prevent this, when returning to the left, it is necessary for the hand to return through a position that is not detected by the sensor. If such an operation had to be performed, it would be extremely troublesome to repeatedly input a gesture in the same direction.

<Move, Return, and Direction Input> A direction gesture input method for solving the problem is shown in FIG. This is a flowchart of a process for stably acquiring a direction gesture.

First, when the user puts his hand in front of the camera sensor and immediately detects the motion information, a malfunction may occur. Therefore, first, the user puts his hand in front of the camera sensor and stops, thereby notifying that the direction gesture is about to be input. The apparatus determines that the user intends to input a direction gesture by detecting that the object has stopped for a predetermined time in a predetermined area in the center of the screen (S23, S24), and starts detecting the direction gesture. .

The trajectory of the hand position is tracked, and it is detected that the hand has moved by a certain amount in any direction (S25, S2).
6). The number of directions to be detected is, for example, four for gesture detection in four directions, and eight for oblique directions in eight directions. Further, there is a case where the motion is detected not only in the vertical and horizontal directions but also in the depth direction.

If it is detected that the object has moved by a certain amount in a certain direction, it is monitored whether or not it moves by a certain amount in the opposite direction, that is, in the returning direction (S27). Then, it is moved by a fixed amount in the opposite direction (S28), and when it is detected that it has stopped still for a certain period of time (S29, S30), it is determined that there is a direction gesture (S31). The "stationary" state described here is not strictly stationary,
Including when the movement is small.

The advantage of this method is that when gestures in the same direction are successively input, the returning operation during that time is not erroneously recognized as a gesture in the opposite direction. This is because the gesture always starts near the center and ends by returning to the vicinity.

Further, when it is desired to input direction gestures one after another, there is an advantage that the operation is easy for the operator because the connection of each gesture is smooth.

<Sound Feedback> Also, when inputting a directional gesture, it is effective to feed back the sound and inform the operator that the input has been accepted. In the case of FIG. 8, a sound is generated at the time when the detection of the direction gesture is completed in the final stationary state. Further, when different sounds are generated depending on the input direction, the user can surely know that the correct input has been performed.

<Feedback with Second Sound> When sound is fed back, if sound is generated in two stages,
May be even more effective. In FIG. 8, the first sound is emitted at the time when it is detected that a certain amount of movement has been made in any direction, the sound is returned to standstill, and the second sound is emitted when the direction gesture is detected.

At this time, the sound that has moved in any direction and the sound that returns when the direction gesture is detected are changed. Then, even if the returning operation is erroneously recognized as an operation moving in one of the first directions, the user can immediately recognize the operation. The first sound may be different depending on the direction, but the returning sound should be the same in any case.

For example, the first sound (up, down, left, right)
If the "pi", "pu", "pe", and the second sound are all "po", the user will receive the top, bottom, left and right sound feedback,
You will hear "papo", "pipo", "popo" and "pepo". If the operation can be performed smoothly, each of these sounds like a single sound effect, and the user does not feel uncomfortable.

<Cancel if not returned> After moving a certain amount in any direction, or when trying to move, if you notice an error or change your mind, you may want to cancel. If the above method is adopted, if the user wants to cancel, he can do so by putting his hand out of the field of view of the camera with the movement.

Alternatively, the movement can be stopped halfway, thereby canceling the movement. In the flowchart of FIG. 8, the state returns to the initial state after a lapse of a predetermined time.

<Return gesture> In the above example,
Although only the direction gesture has been described, in general, when information is presented by a position trajectory, a series of gestures "starting from a certain area, reaching and ending in that area" is a series of gestures Very suitable. For example, the gestures shown in FIG. 9 are all the same, so that not only the direction but also various gestures can be input. In FIG. 9, an arrow indicates a locus of position.

<Excluding Movement When Returning Due to Difference in Distance> With the above-described configuration, it is possible to present a continuous direction gesture while preventing the returning operation from causing a malfunction. However, depending on the case, there is also a case where it is desired to make a directional gesture by a movement that moves in one direction. When performing such a return movement, it is often the case that the hand is slightly pulled back.

For example, assuming that the user strokes twice on the desk to the right, first stroke the surface of the desk once to the right, lift the hand slightly from the desk, return to the left, and stroke again to the right.
This operation is normally performed. The “slightly lifted state” is a “slightly pulled down state” in consideration of a gesture in the air.

By the way, the image input device described in JP-A-10-177449 can acquire relative distance information of a hand. Therefore, it is possible to detect that the movement to return to the left is performed in a "slightly pulled state" rather than to the movement to the right. FIG. 10 shows a configuration for performing this.

The image input from the image input unit 32 is input to the motion detecting unit 34 and the distance detecting unit 33, and the motion information 39 and the distance information 38 are obtained respectively. Judgment unit 35
Determines from the information whether the motion information input from the motion detection unit 34 is a motion input or an operation for returning.

More specifically, if a motion in the opposite direction is detected immediately after a direction gesture in a certain direction is detected, and the distance information at that time is farther than the previous direction gesture, this returns. It is determined that the operation is performed.
If the gesture is determined to be a direction gesture, the information of the direction gesture is transmitted to the operation execution unit 36, and feedback is generated by sound as needed. If it is determined that the movement is a return movement, nothing is transmitted to the operation execution unit, and no sound feedback is generated.

Regarding the sound feedback, if the sound is changed on the right and left so that the return does not generate a sound, the user immediately notices that the return operation is erroneously detected as a direction gesture. be able to.

<Judge the Second Movement as a Return Operation>
Such processing is not necessarily impossible even if an image sensor that cannot acquire distance information is used. In the case of an image sensor that can only acquire motion information, it is possible to determine the return operation by looking at the time series of the motion.

For example, “right”, “left”, “right”, “left”, “right”
Is input three times in the right direction, and it is determined that the two left gestures in the middle are a return operation. However, this is not enough to change the left gesture on the way. It can be determined by making good use of the time interval between each movement, so-called “ma”.

For example, “right”, “left”, “right” (between), “left”
If "Right" or "Left" is entered, "Right" or "Right"
It can be determined that "left" and "left" have been input after the input of.

<Following the Menu Hierarchy> So far, the gesture for presenting a shape and the gesture for inputting a direction have been described, but it is possible to follow the menu hierarchy using these gestures. . Interfaces for operating a device having a large number of functions often have functions that are divided into categories, have a hierarchical structure, and reach functions by tracing from top to bottom.

For example, a computer often adopts a method of sequentially selecting menus on a screen while moving a cursor with a mouse. For a vehicle-mounted device such as a car navigation system, a cross button of a touch panel or a remote control is used. In many cases, a menu is sequentially selected.

When selecting a menu, it is an effective method to input a direction gesture without using a direction key or the like, or to present a shape gesture instead of pressing a function key. FIG. 11 shows a configuration diagram thereof. The image input from the image input unit 40 is processed by the gesture detection unit 41, and when a gesture is presented, it is detected. The menu control unit 42 determines how to follow the menu based on the detected gesture, and causes the menu display unit 43 to display a new menu.

When a gesture is detected, the feedback section 44 gives feedback to the user by sound or voice. As an advantage at this time, since the operation is a gesture, it is easy to memorize the operation until reaching a certain function. For example, if the shape gesture is used to indicate “goo”, “choki”, and “par”, the temperature of the air conditioner is set. The user does not need to memorize all of these, but only the frequently used ones in the time series of gestures.

Actually, the time series of gestures is still a gesture, and in this respect, it can be said that this interface associates all functions with gestures on a one-to-one basis. At this time, if the same shape continues, it is 2
It's hard to show that it's happening many times. Therefore, "par"
Is displayed and the menu moves to the next menu hierarchy, it is convenient that there is no menu presenting “par” in that hierarchy. That is, no matter which function is selected, the menu gesture should be configured so that the same gesture is not presented twice.

<Follow the menu hierarchy with direction gesture>
Although the method of tracing the menu hierarchy by presenting the shape gesture has been described, it is similarly possible to use the direction gesture. When the directional gesture is used, it is relatively desirable to use the four-directional gesture in consideration of the easiness of movement of the human hand. However, the four-directional gesture has a disadvantage that only four menu options can be displayed. There are several ways to solve this.

<Repeated input> For example, "once to the right"
It is sufficient to configure so that different options can be selected when the user moves only the button and when the user moves “twice to the right”. this is,
Of the above-described direction gesture input methods, a series of operations of moving a hand in any direction and returning to the original position are suitable for use as a direction gesture.

For example, if the user moves rightward by a fixed amount and moves to the original position and stops, "Right" is input. However, if the user does not stop and moves to the right again and returns to the original position and stops, "Right 2" is input.

If this is repeated, basically an infinite number of options can be selected, but in reality, it is appropriate to limit the repetition to about two or three times. This is about two mouse clicks, usually a single click and a double click, and some professional computers have realized a triple click. Conversely, double clicks are used by ordinary users as a matter of course, and there are no problems with continuous input about twice.

When traversing the menu hierarchy using the direction gesture, a menu screen as shown in FIG. 12, for example, can be considered. Since each menu is arranged in the direction of movement, the configuration is easy to see.

In the case where two consecutive inputs are permitted, FIG.
A menu screen as shown in FIG. The inner menu is selected by one direction input, and the outer menu is selected by two direction menus. Also, if three consecutive inputs are allowed, the ten-key can also be input by a directional gesture (FIG. 13B).

<Shape + Direction> When inputting a direction gesture, there is also a method of changing the shape of the hand and selecting an option by a combination of the shape and the direction. If the direction is determined by, for example, moving the center of gravity of the entire hand, the direction can be presented regardless of the shape of the hand. At this time, different options can be selected when moving the finger to the left with one finger extended and when moving the finger to the left with the two fingers extended. It takes time and effort to input the direction repeatedly, but when the shape and the direction are combined, it is relatively easy to increase the number of options.

<Start Instruction Button> Detecting a gesture requires some processing cost, and there is always a risk that casual actions may be picked up as gestures. These problems do not occur if the gesture presentation button is always pressed when the gesture is presented.

<Selection by Two-Dimensional Pointing> By mounting such an input interface using a gesture on, for example, an in-vehicle device, a safe operation during driving can be realized. However, some of these operations are not operated during driving, and the line of sight does not necessarily need to be fixed forward when operated by a passenger.

Therefore, operations that are rational to be performed while looking at the screen may be realized as such. Instructing by direction has the merit that you can operate without directing your gaze, but when choosing from many options,
As mentioned earlier, presenting repeated direction gestures,
It was necessary to use both shape and direction. In a state where the line of sight can be directed to the screen, a method of selecting using the position of the point may be used.

At that time, a problem is the selection decision input. For example, when selecting an icon on the screen using a mouse, the user moves the mouse cursor to an icon and then clicks the icon to select it. In the case of using a gesture to achieve the same thing, the problem is how to make a decision input corresponding to a click. A possible method is to change the shape of the hand to “grab” from par to goo, or to perform a “push” action to handle a button, and use that as a decision input.

However, when such an operation is performed, the position information fluctuates in the course of the operation, and the icon next to the selected icon is often selected. this is,
This is because the operation of "moving the position" and the operation of "changing the shape" or "pushing back and forth" are not independent. This may be difficult for a person to control them independently, or it may be difficult to separate them as a property of the device.

Further, a selection trigger can be used in a stationary state for a certain period of time or more. However, there is a case where a period of time for standing still is required or the stationary state is reached while the user is lost. It is easy to move independently in human motion, and it can be sensed independently.
By using the two movements for “selection” and “decision,” a more stable operation can be realized.

This will be described with reference to FIG. Image input unit 46
The input image is transmitted to the first motion detecting section 47 and the second motion detecting section 49, and each detecting section detects independent motion information as described above. The option or value selection unit 48 determines what is currently selected according to the output of the first motion detection unit 47. This result is typically fed back to the operator by, for example, inverting display on a display unit (not shown).

The decision operation determination section 50 analyzes the output of the second motion detection section 49 and determines whether a decision operation has been performed. The operation input processing unit 51 uses the output of the option or value selection unit 48 and the determination operation determination unit 50 to determine what has been selected and determined, and actually performs processing.
Send to 2.

<Selection and Determination with Two Movements> More specifically, for example, a method of combining left and right movement and up and down movement is used. Use left and right movements to select menus arranged side by side. The currently selected menu is highlighted, for example, so that the user can know what is selected. In this state, this is a tentatively selected stage, from which the options are determined by moving up and down.

In the above example, "selection" is made in the left-right direction and "decision" is made in the up-down direction, but there is a reason for this. In general, a person can specify a position more finely by a movement in the left-right direction. Therefore, the left-right direction is defined as a “selection” direction.
Since the vertical movement is only a trigger input, there is no problem even if it cannot be controlled so finely. However, this is general,
The opposite may occur depending on the operation conditions (body posture, etc.), but in any case, select the direction that is easy to control finely.
It is important to make the direction.

For example, FIG. 15A shows an example of a menu screen in the case of making a selection by this method. As shown by an arrow in the figure, a selection is made by a horizontal motion and a vertical motion is selected. Determined by In FIG. 11B, since there are many options, icons for scrolling left and right are arranged. When this is selected, the menu scrolls one after another.

<Using the Up and Down Movements Separately> In the above example, both the up and down directions are simply “determined”.
It is also possible to arrange the menu up and down in two steps, and to select the upper and lower menus respectively in the upper and lower directions. This allows you to see twice as many choices at once. In this case, the tentative selection state (for example, reverse display) is determined by selecting two menus at the top and bottom simultaneously and inputting "OK".

FIG. 16 shows an example of a menu screen for performing this selection method. For example, Ibaraki Prefecture and Yamanashi Prefecture are selected, and if moved up, Ibaraki Prefecture is selected, and if moved down, Yamanashi Prefecture is selected. Here, the icons of “other areas” and “return” at the end are selected by moving either up or down.

<Move up and down and return> In the examples so far, if the user moves up or down to select, the hand moves up or down, so it is not suitable for performing a continuous menu operation. . If it is determined by moving a certain amount up or down and returning to the original place, continuous menu selection can be easily performed. Just like moving to the right and recognizing a “right” gesture when returning to the center in the previous example.

<Horizontal, Vertical, Horizontal and Continuity> When characters are selected from the Japanese syllabary, there are many options to be displayed at one time. In such a case, first, use the horizontal movement to
A method of selecting "line" such as "line" or "line", then selecting one of the five characters by a vertical movement, and further selecting by a horizontal movement. Also, once you select a line by horizontal movement, and then select a character by the next vertical movement,
Even if it moves a little in the horizontal direction, it should not be appropriate to select a different line again.

In this case, once the horizontal direction is determined, it is preferable to perform processing such that no horizontal movement is detected until the next vertical direction is determined. Of course, in the case of the Japanese syllabary table, the cursor can be simply displayed at the upper, lower, left, and right positions and directly selected. In this case, the motion in the depth direction is determined based on still information or the like. Furthermore, when the depth is changed, it is possible to use the depth information further, for example, to change to hiragana or katakana.

Further, a cloud point, a semi-voice point and the like can be designated by the shape of the finger when pointing, which is more efficient than inputting the cloud point and the like independently. For example, if you select it with two fingers,
If you select it with your finger, it will be a semi-voiced point.

<Sensor mounting position>
Using the image input sensor and the like proposed in 177449,
When a vehicle-mounted operation input device is configured, the installation position of the sensor greatly affects usability and performance.

If it is of the utmost importance that the driver can operate safely while driving, place the camera in a place where the driver can operate with the hand on the steering wheel or with almost no release from the steering wheel. Good to do. For example, it is an example to arrange on the back side of the handle so as to face the hand holding the handle.

[0146] Further, although a normal operation is not performed when turning a curve, a case where an operation is performed when the steering wheel is slightly turned, such as a gentle curve, can be assumed. Then, it can be attached directly to the handle on the back side of the handle. Then, regardless of the state of the steering wheel, there is no change in the positional relationship (distance, direction) between the hand and the camera,
Stable detection performance can be obtained.

In these cases, since the image input often includes components such as handles, processing is performed so as not to be affected by these components. Also, at this time, when processing the image, it is easier to determine the vertical and horizontal movement of the image than to detect the movement in the oblique direction.
It is important to set the orientation of the camera according to the position of the hand and the direction of its movement.

In the case where not only the driver but also the passenger are operating, it is desirable to arrange them at an intermediate position between them.
In this case, it is necessary to be arranged at a position where it is possible to detect a place where the driver can easily reach.
In addition, when performing a gesture operation, particularly when pointing a position, the operation can be stably performed by fixing a part of an arm such as an elbow or a wrist or placing the arm on a table or the like.

Therefore, it is important to place a base on which an elbow or a wrist can be placed at an appropriate place. This is also responsible for determining the approximate location of the operating hand,
It is important to ensure stable performance. For example, a shape like 53 in FIG. 17 can be considered. Here, the camera 54 is arranged in an oblique direction, and a recess is provided so that the hand does not approach the camera very much.

The image input device invented in Japanese Patent Application Laid-Open No. H10-177449 has its own light emitting means.
The object shape can be imaged regardless of the surrounding brightness. However, if there is extremely bright external light, such as direct sunlight, it may be affected. In particular, if strong external light is applied to a portion of the image input device that is illuminated by the light emitting means, the image is likely to deteriorate.

Therefore, arranging the sensor in such a positional relationship that strong external light does not hit the surface of the hand picked up by the image input apparatus can improve the performance and reduce the cost of the optical BPF used. There is.

For example, FIG. 17B shows a cross-sectional view of the shape of FIG. 17A. Since the palm 55 faces downward, strong external light rarely hits it. It is also effective to have a means for shielding external light to such an extent that the operability is not impaired.

<Recognition of Drivers and Passengers> Advanced information devices such as car navigation are dangerous when the driver concentrates on operations, and there are operations that must not be performed during driving. For example, the operation of registering a destination is very complicated, and cannot be performed during driving. Therefore, normally, the input of the vehicle speed sensor is used, and these operations cannot be performed during traveling.

[0154] However, even while the vehicle is running, the operation by the passenger is not inherently dangerous, but at present, the operation cannot be performed. If it is possible to detect whether the person who is currently performing the operation is a driver or a passenger, it is possible to prohibit the operation of only the driver. By using the image sensor to image the entire operating person or, if not all, the upper arm of the operating hand, the hand extending from either the driver's seat or the passenger seat Can be determined.

FIG. 18 is a block diagram of an apparatus for realizing this. The operator determining unit 58 determines whether the operator is a driver or a passenger using the input image from the image input sensor 57. The result is sent to the operation input processing unit 60, and if the function cannot be operated, the execution is refused and the operator is notified that the function cannot be operated.

<Combined Use with Voice Recognition> An input device using voice recognition is a device that can be safely operated by a driver while driving, as in the case of input by gesture described in the present invention. In this method, a voice emitted by a driver is detected by a microphone, the voice data is analyzed, a user's intention is determined, and the result is linked to an operation. Devices using voice recognition have the advantage that they can be used by anyone because they can easily remember commands and speak, but also have the problem that there are many erroneous recognitions that must always be made when operating. By combining the gesture input device of the present invention with speech recognition, a system that is easier to handle can be constructed.

[0157] For example, the function can be selected immediately without tracing the hierarchy by using voice. Therefore, voice recognition is used, and parameter input is performed using intuitive gestures. For example, after "volume", the volume is dynamically adjusted by moving the hand left and right (or up and down or in the depth direction). When it is just right, it is decided to say "OK".

Alternatively, a voice input can be used as a determination input when designating a position. Furthermore, a plurality of executable operations can be instructed by voice input to the target selected by the gesture.

For example, a restaurant displayed on a map can be selected by a gesture and a voice command such as “telephone”, “information”, and “route” can be issued. When "phone" is spoken, the restaurant is called, when "information" is spoken, information about the restaurant is displayed, and when "route" is spoken, a route search to the restaurant is started.

<Contact plate> It has already been described that a more stable operation can be performed when the elbow, the wrist, or the like is put on a table or the like, but a table or the like on which the hand or finger to be detected can be put is prepared. Also has a great effect. Such a configuration is shown in FIG. Here, JP-A-10-17
The image input device described in 7449 is arranged as an image sensor.

Further, a transparent or translucent plate 56 is disposed between the operator and the hand. If anti-reflection processing is applied to this, extra reflection is eliminated and an image can be stably obtained. Alternatively, the plate may have an optical band-pass characteristic such that only the wavelength of light emitted by the light emitting means included in the image input device is transmitted. In the case of a translucent plate, the amount of light passing therethrough changes, so that the operation is basically the same as when there is no plate, except for affecting the sensitivity. The effect of the presence of the plate is as follows.

First, the operator always performs the operation by touching or not touching the plate, so that the distance between the hand and the sensor camera can be kept almost constant, and the detection can be stably performed. is there. Also, this is particularly remarkable when indicating a position, but it is easier for a person to rest on a plate than to keep his finger still in the air.

When inputting a direction gesture,
By slightly touching the board when moving in the direction instructed and not touching when returning, the difference in distance can be reliably represented. Further, the detection distance when touching the plate is fairly stable and fits within a small width, so that it is possible to determine whether or not the touch is made by a distance value (actually, distance information converted from the amount of reflected light). .

Thus, for example, the shape gesture can be represented by the number of fingers touching the board with one finger, touching the board with two fingers, and the like. Further, since the shape of the entire hand is also detected, it is possible to detect which finger is touched, and it is possible to easily increase the types of shape gestures.

[0165] In this figure, the position of the camera captures the hand obliquely in the finger direction, but this position is not limited to this. For example, the camera position may be right under the hand or obliquely in the wrist direction. May be. However, since the shape of the fingertip is often somewhat important, it is better to take an image from this direction.
Stable detection becomes possible.

<Shape Detection> If it is desired to operate only by touching the shape, a different structure can be provided. As an image sensor, a CCD camera is provided, and there is a diffusion plate for diffusing light from the outside. Normally, light from the outside enters the camera through the diffuser,
The camera always captures dimly bright images. When the operator places his or her hand or finger on the diffusion plate, light does not reach from that part, and thus the image is taken as a black part.

By analyzing this image, it is possible to detect how many fingers have been touched, or which direction the touched finger has moved. Alternatively, if not touched but very close, the image is captured as gray close to black, so that this can also be used.
When the surroundings are dark, external light cannot be used. In this case, a similar operation can be performed by providing an auxiliary light source outside.

Further, the processing in the embodiment of the present invention can be realized by a computer-executable program, and the program can be realized as a computer-readable storage medium.

<Recording Medium> The storage medium in the present invention includes a magnetic disk, a floppy disk,
Hard disk, optical disk (CD-ROM, CD-
R, DVD, etc.), a magneto-optical disk (MO, etc.), a semiconductor memory, or any other storage medium that can store programs and that can be read by a computer.

An OS (operation system) running on the computer, database management software, MW (middleware) such as a network, etc., according to the instructions of the program installed in the computer from the storage medium, realize this embodiment. May be executed.

Further, the storage medium in the present invention is not limited to a medium independent of a computer, but also includes a storage medium in which a program transmitted through a LAN, the Internet, or the like is downloaded and stored or temporarily stored.

Further, the number of storage media is not limited to one, and the case where the processing in the present embodiment is executed from a plurality of media is also included in the storage media of the present invention. Good.

The computer according to the present invention is:
The computer executes each process in the present embodiment based on a program stored in a storage medium, and may have any configuration such as an apparatus such as a personal computer or a system in which a plurality of apparatuses are connected to a network. Is also good.

The computer in the present invention is not limited to a personal computer, but also includes an arithmetic processing unit, a microcomputer, and the like included in information processing equipment, and generically refers to equipment and devices capable of realizing the functions of the present invention by a program. are doing.

<Summary> According to the first aspect of the present invention, the operation is performed using two different types of gestures among the gestures such as the shape and movement of the human body or a part thereof, and the first type of gesture is performed. Thus, it is possible to realize an operation input device that can select an operation mode and change parameters without the operation mode using the second type of gesture.

According to the second aspect of the present invention, it is possible to realize an operation input device which is extremely natural for a user, in which an operation mode is selected by a shape gesture of presenting a shape, and parameters are changed by a direction gesture indicating a direction.

According to the third invention, one or a plurality of similar operation modes are associated with one shape gesture, and when the shape gesture detection means continuously detects the same shape gesture, Since it is possible to sequentially select a plurality of operation modes associated with the shape gesture, when there are a plurality of similar operation modes, assigning different gestures to all of them causes a large burden on the user. Many similar operating modes can be selected without adoption.

According to the fourth aspect, the shape gesture detecting means starts the shape detection when detecting that the motion of the human body or a part thereof is stationary or has a small motion for a predetermined time or more. Therefore, it is possible to prevent an unintended movement of the operator from being erroneously recognized as a shape gesture. Further, since the shape recognition processing is performed only when the apparatus is stationary, processing costs can be reduced.

According to the fifth aspect, an image representing a selectable operation mode and a shape gesture for selecting the operation mode or an image representing a changeable parameter and a direction for changing the selectable operation mode is displayed on the display means. Since the display is also performed, when the operator forgets the gesture, the type of the gesture can be known without having to open the instruction manual.

According to the sixth aspect, information such as a currently selectable operation mode or a changeable parameter is presented by voice, so that the operator can know the current operation status while gazing forward. Can be.

According to the seventh aspect, voice presentation is performed when the gesture presentation by the operator is delayed during the operation, so that there is no trouble that the voice presentation is always performed.

According to the eighth aspect of the present invention, there are some of the following: the shape gesture detection means has started detection, the detection has been completed, the detection has failed, and the direction gesture detection means has detected the direction. Or, when everything happens, it has a sound presentation means that presents to the operator with sound, so that whether or not the gesture made by himself has been correctly recognized,
You can know without looking at the screen.

According to the ninth aspect, since different sounds are presented depending on the type of operation or the type of direction detected, when a gesture different from the direction or shape to be input is detected. You can know.

According to the tenth aspect, the direction gesture is detected by analyzing the trajectory of the position information corresponding to the position of the human body or a part thereof, the one held by the person, or the one worn by the person. The object moves in a predetermined direction in one of a plurality of predetermined directions, and then moves in the opposite direction by a predetermined range. By using the direction gesture detection method, characterized by detecting as a direction gesture in the direction of the first movement, when a gesture in the same direction is repeatedly presented, the "return" operation occurring between the opposite directions is performed. Can be prevented from being erroneously recognized as a gesture. In addition, since the gesture is always returned to the same place and the gesture ends, it is suitable for repeating the direction gesture.

According to the eleventh invention, the direction gesture is detected by analyzing the trajectory of the position information corresponding to the position of the body or a part of the person, the object held by the person, or the object worn by the person. In the method, after the object stays in a stationary or low-motion state for a predetermined range of time in a predetermined range, in one of a plurality of predetermined directions, Moves in a fixed range, then moves in the opposite direction by a fixed range, and then remains stationary or low-motion for a fixed range of time again, in the direction of the first move By using a direction gesture detection method characterized by detecting as a direction gesture, it is possible to prevent an unintended movement of the operator from being erroneously detected as a direction gesture. Door can be.

According to the twelfth aspect, when the direction gesture is detected, the presence of the sound is provided by sound, so that it is possible to confirm that the direction gesture has been correctly detected.

According to the thirteenth aspect, since different sounds are presented depending on the direction of the detected gesture, it is possible to notice when a direction different from the input direction is erroneously detected.

According to the fourteenth aspect, when one moves in a certain direction at first, one sound is presented, and when moving in the opposite direction, another sound is presented when detected as a direction gesture. It is possible to reliably execute the initial movement and the return movement.

According to the fifteenth aspect, the first sound is:
Different sounds are presented depending on the direction of movement, and the second sound presents the same sound regardless of the direction gesture, so that the same sound is always heard when the direction gesture is completed. Therefore, when a situation in which the initial movement and the return movement are detected in reverse is immediately noticeable.

According to the sixteenth aspect, a gesture is detected by analyzing the trajectory of position information corresponding to the position of a person's body, a part thereof, a person's possession, or a person's body. A method is used, wherein a gesture detection method is characterized in that a motion intentionally moved by a person is distinguished from a motion that occurs unintentionally. Thus, when performing a gesture in the air, an operation that is intentionally performed by a person and an unintended operation that occurs between the intended operations are identified, and an operation that is effective for determination is performed. Only can be taken out.

According to the seventeenth aspect, the direction gesture is detected by analyzing the trajectory of the position information corresponding to the position of the human body or a part thereof, the object held by the person, or the object worn by the person. Means for detecting movement in any one of a plurality of predetermined directions, and means for removing a "return" operation occurring between successive same-direction gestures. By using the directional gesture detection method having the directional gesture, it is possible to prevent erroneous detection of a “return” operation, which is a problem when a directional gesture is indicated, in one-directional movement.

According to the eighteenth aspect, there is provided a means for detecting a distance from the sensor of a movement, and when a movement in a direction opposite to the distance is detected during a continuous movement in a certain direction, the movement is detected. Since it is determined that the movement is "return" and is removed, when the operator performs the "return" operation, the operator moves his hand away from the camera sensor and returns it.
Operation can be eliminated.

According to the nineteenth aspect, there is provided a means for measuring a time interval between each movement, wherein a movement in an opposite direction is detected during a continuous movement in a certain direction, and between the movements. If there is only a time interval equal to or less than the predetermined length, the motion during this time is determined to be a “return” motion and removed, so when presenting the same direction gesture, make sure that the time interval is not empty When a gesture in the reverse direction is presented, the operation is stopped and then performed again, so that a stable operation without the "return" operation can be performed.

According to the twentieth aspect, since different sounds are presented depending on the directions for the detected direction gestures, it is possible to know when the direction gesture is recognized without removing the “return” operation. be able to.

According to the twenty-first aspect, a trajectory of position information corresponding to the position of a person's body, a part thereof, a person's possession, or a person's wear is analyzed and detected as a gesture. This method is suitable for repeatedly inputting various gestures because a trajectory in which the start position and the end position are close to each other is detected as a gesture.

According to the twenty-second aspect, an operation input device using the above-described direction gesture detection method can be realized.

According to the twenty-third aspect, a shape gesture detecting means capable of detecting a shape of a human body or a part thereof, a menu displaying means for displaying a menu, and a detected gesture are associated with each other. An input device having menu selection means for selecting a menu, wherein a gesture is associated with a menu so that a shape gesture having the same shape is not presented when the menu is hierarchically traversed. By realizing the operation input device characterized by the above feature, it is possible to perform menu selection across hierarchies by combining gestures as if a single gesture is presented.

According to the twenty-fourth aspect, a direction gesture detecting means capable of detecting a direction of movement of a human body or a part thereof, a menu displaying means for displaying a menu, and a function corresponding to the detected gesture In addition, an input device having menu selection means for selecting a menu, wherein when a direction gesture is continuously presented, a menu different from that presented only once is selected, so that the direction gesture is simply performed. Many menus can also be displayed when using gestures.

According to the twenty-fifth aspect, a shape gesture detecting means capable of detecting the shape of a human body or a part thereof, and a direction of movement of the human body or a part thereof can be detected. An input device having a direction gesture detection unit, a menu display unit for displaying a menu, and a menu selection unit for selecting a menu in association with the detected gesture, wherein the detected direction gesture is detected simultaneously with the detected direction gesture. By selecting the menu to be selected in accordance with the combination with the shape gesture, many menus can be displayed even when simply using the directional gesture.

According to the twenty-sixth aspect, one of the arranged items is selected using information on the position and movement of a person's body or a part thereof, a thing held by a person, or a thing worn by a person. Using two types of position and motion information that can be easily and independently controlled by a person or that can be easily and independently detected by a detecting means. An operation input method is characterized in that an item or a value is selected and the other is determined using the other. Thus, when the screen can be viewed, an item or a value can be input quickly. These two types of movements are highly independent, affecting each other's movements,
It is possible to prevent a detection error from occurring.

According to the twenty-seventh aspect, the item or value is selected by moving in one direction, and the motion is determined by presenting the motion in a direction different from that, so that the selecting operation can be performed only with the motion information. The operation corresponding to the click need not be indicated by a change in shape or the like, so that the operation can be performed more stably.

According to the twenty-eighth aspect, the direction in which a person can control the position more finely or the direction in which the detecting means can detect the position with higher accuracy is used as the direction of selection, so that the operation can be performed with higher accuracy. In addition, it is possible to provide a device that is easy for the user to operate.

According to the twenty-ninth aspect, when presenting the determination direction gesture, different items or values are selected and determined according to two opposite directions, so that more items are displayed and selected. It is possible to do.

According to the thirtieth aspect, when presenting the determination direction gesture, when the user moves in the determination direction by the amount of the predetermined range and returns again by the amount of the predetermined range. In addition, since the item is determined, the operation can be efficiently performed when this item selection is repeatedly performed.

According to the thirty-first aspect, an operation input device using the operation input method according to the twenty-sixth to thirtieth aspects can be realized.

According to the thirty-second aspect, by providing a place for placing a part of an arm, such as an elbow or a wrist, more stably,
Operation can be performed easily.

According to the thirty-third aspect, the first to ninth, twelfth to fifteenth, twentieth, twenty-second to twenty-fifth, and twenty-fifth to twenty-fifth, the position and the shape of the target object are detected using optical means. 31. The operation input device according to the thirty-first aspect, wherein the operation input device is devised in a direction and arrangement such that strong external light does not hit an object surface on the image input device side. Stable performance can be exhibited even by using optical means that is easily affected by external light.

According to the thirty-fourth aspect, the first to ninth, twelfth to fifteenth, twentieth, and twenty-second to twenty-fourth units are used in vehicle-mounted devices.
An operation input device according to the twenty-fifth and thirty-first inventions, wherein the operation input device is arranged so that the driver can operate the steering wheel without releasing his hand from the steering wheel or almost without releasing it. Thus, the driver can operate more safely while driving.

According to the thirty-fifth aspect, the first to ninth, first
An operation input device according to any of the second to fifteenth, twentieth, twenty-second to twenty-fifth, and thirty-first inventions, comprising an operator determination unit configured to determine to which category a person who intends to operate belongs. The operation input device is characterized in that the operation input device is characterized by having means capable of changing the operation that can be performed depending on the category to which the operator belongs. Accordingly, it is possible to permit or prohibit the operation according to conditions such as the state and position of the user.

According to the thirty-sixth aspect, the first to ninth, twelfth to fifteenth, twentieth, and twenty-second to twenty-fourth units are used for in-vehicle devices.
The operation input device according to the twenty-fifth and thirty-first inventions, further comprising: an operator determination unit configured to determine whether an operation is to be performed by a driver or another driver. It is characterized by having a means to control so that some operations can not be performed under the conditions of the above, so that even if the driver should not operate while driving, it can be operated by the passenger it can.

According to the thirty-seventh aspect, the first to ninth, first to ninth
The operation input device according to any one of the second to fifteenth, twentieth, twenty-second to twenty-fifth, and thirty-first inventions, further comprising a voice input unit capable of giving an instruction by voice, so that a simple input by voice is performed. An environment that is easier to operate can be realized by using both a method and gesture input suitable for parameter input.

According to the thirty-eighth aspect, the first to ninth, first to ninth,
The operation input device according to any one of the second to fifteenth, twentieth, twenty-second to twenty-fifth, and thirty-first inventions, further including a stabilizing unit for preventing unintended movement of a user and stabilizing movement of a hand or body. By using the operation input device characterized by the above, more stable operation can be realized.

According to a thirty-ninth aspect, the stabilizing means is a contact plate for allowing a user to make contact with a hand or a body at the beginning or end of an operation or during an operation to operate. By using the operation input device of the invention of the invention, the operator can input the position more stably by touching the board, and can feel reassured that the operation is performed by touching the board. Further, since the distance is kept constant, more stable detection performance can be realized.

According to the fortieth aspect, a contact shape detecting means capable of detecting a shape of a portion touched by a hand, a shape gesture detecting means detecting the contact shape as a shape gesture, and a direction of movement are detected. According to the detection result of the direction gesture detection means, the operation mode selection means for selecting one of a plurality of operation modes in which a specific operation can be performed according to the detection result of the shape gesture detection means, and the detection result of the direction gesture detection means By configuring an operation input device having a parameter changing unit that changes a parameter that can be changed in the operation mode, it is possible to realize an operation input environment that operates only when touched and operated reliably. With this configuration, a stable operation can be realized even with a normal CCD camera.

According to the forty-first aspect, the first to ninth, first
An operation input device according to any one of the second to fifteenth, twentieth, twenty-second to twenty-fifth, and thirty-first inventions, wherein the light-emitting device emits pulsed light and operates in synchronization with the light-emitting device. By using the operation input device having a means for imaging only the object reflected light, the shape of the object can be obtained stably and at high speed, so that gesture input can be realized at low processing cost.

According to the forty-second aspect, the first to ninth, first
2nd to 15th, 20th, 22nd to 25th, 31st to 41st
A vehicle equipped with the operation input device of the invention of the invention can be realized.

According to the forty-third aspect, shape / movement information of an object such as a human body or a part thereof or an object is detected, and one of a plurality of operation modes is selected based on the detection result. Then, using the operation input method characterized by detecting different types of shape / movement information of the same object, and changing the changeable parameters in the operation mode using the detection result. A device that can be operated by a person with a natural feeling can be configured.

According to the forty-fourth aspect, the tenth, eleventh,
The storage medium storing the program for executing the gesture input method or the operation input method according to the sixteenth to nineteenth, twenty-first, twenty-sixth, and thirty-third inventions is stored in an apparatus in which a computer is mounted. The operation input environment as described above can be constructed.

[0219]

According to the present invention, two different gestures, such as the shape and movement of the human body or a part thereof, are used.
An operation input device capable of performing an operation using two types of gestures, selecting an operation mode with a first type of gesture, and changing a parameter within the operation mode with a second type of gesture Can be realized.

[Brief description of the drawings]

FIG. 1 is an overall configuration diagram for realizing the present invention.

FIG. 2 is a more specific overall configuration diagram for realizing the present invention.

FIG. 3 is an example of a shape gesture used in the present invention.

FIG. 4 is a transition diagram of an operation using a shape gesture and a direction gesture used in the present invention.

FIG. 5 is a display example of a gesture icon used in the present invention.

FIG. 6 is a configuration diagram for performing audio feedback used in the present invention.

FIG. 7 is a flowchart of processing for performing audio feedback when an operation used in the present invention is delayed.

FIG. 8 is a flowchart showing a direction gesture detection method used in the present invention.

FIG. 9 is an example of a gesture whose start and end positions are close to each other used in the present invention.

FIG. 10 is a configuration example for detecting a distance used in the present invention and eliminating a return operation.

FIG. 11 is a configuration example of an apparatus that can follow a menu hierarchy by using a gesture used in the present invention.

FIG. 12 is an example of a screen when a menu is selected by a direction gesture used in the present invention.

FIG. 13 is an example of a screen for selecting a different menu by repeating a direction gesture used in the present invention.

FIG. 14 is a configuration diagram of an apparatus that performs selection and determination using two types of independent motion information used in the present invention.

FIG. 15 is an example of a screen of a device that performs selection and determination using two types of independent motion information used in the present invention.

FIG. 16 is an example of a screen in a configuration in which a motion for determining is further divided and the number of options is increased.

FIG. 17 is an example of the appearance of an apparatus used in the present invention.

FIG. 18 is a configuration diagram of a device that determines an operator and adaptively permits / disallows an operation.

[Explanation of symbols]

 DESCRIPTION OF SYMBOLS 1 ... 1st gesture detection part 2 ... 2nd gesture detection part 3 ... operation type selection part 4 ... parameter change part 5 ... operation input processing part 6 ... operation execution part 7 ... shape gesture detection part 8 ... direction gesture detection part 9 Gesture icon 13 Voice feedback unit 32 Image input unit 33 Distance detection unit 34 Motion detection unit 47 First motion detection unit 48 Selection unit 49 Second motion detection unit 50 Determination operation determination Part 53: A table on which part of the arm is placed 54 ... Camera 56 ... Contact plate 58 ... Operator determination part

 ──────────────────────────────────────────────────続 き Continuing on the front page (72) Inventor, Isao Mihara 1st, Komukai Toshiba-cho, Saiwai-ku, Kawasaki City, Kanagawa Prefecture Inside the Toshiba R & D Center (72) Inventor Takahiro Harashima, Komukai Toshiba, Sai-ku, Kawasaki City, Kanagawa Prefecture No. 1 in the Toshiba R & D Center (72) Inventor Yasunari Yamauchi 1 in Komukai Toshiba-cho in Sachi-ku, Kawasaki-shi, Kanagawa Prefecture In-Toshiba R & D Center (72) Inventor Akira Morishita, Kawasaki-shi, Kanagawa 1F, Komukai Toshiba-cho, Toshiba R & D Center F-term (reference) 5E501 AA20 AA22 AB03 AB06 AC03 BA05 CA02 CA07 CA08 CB14 CC11 CC14 EA33 EB05 FA04 FA05 FA32 FA42

Claims (25)

[Claims] A first detecting means for detecting a shape and a movement of the object to be detected, and a shape and a movement of the object different from the shape and the movement detected by the first detecting means. Second detecting means for detecting; an operation selecting means for selecting a specific operation based on a detection result of the first detecting means; a changeable parameter in the operation selected by the operation selecting means An operation input device, comprising: a parameter changing unit that changes the parameter based on a detection result of the second detection unit. 2. A shape detecting means for detecting a shape of an object to be detected, a direction detecting means for detecting a direction of movement of the object, and a specific operation based on a search result of the shape detecting means. And an operation selecting means for changing the parameter which can be changed in the operation selected by the operation selecting means based on the detection result of the direction detecting means. apparatus. 3. The operation selecting means associates a plurality of operations with the shape of the object, and when the shape detecting means continuously detects the same shape gesture, the operation selecting means is associated with the shape. The operation input device according to claim 2, wherein a plurality of operations are sequentially switched and selected. 4. The apparatus according to claim 2, wherein said shape detecting means detects the shape of the object when detecting that the movement of the object is stationary or in a state of a small movement for a predetermined time or more. An operation input device according to item 1. 5. Display means for displaying information of a selectable operation or a parameter which can be changed, the display means comprising an image showing a selectable operation mode and a shape gesture for selecting the operation mode, 3. The operation input device according to claim 2, wherein a changeable parameter and an image indicating a direction for changing the changeable parameter are displayed together. 6. The operation input device according to claim 2, further comprising a voice presenting unit for presenting information of a selectable operation or a changeable parameter by voice. 7. The operation input device according to claim 6, wherein when the processing of the shape detecting means or the direction detecting means is delayed, the voice presentation is performed. 8. A detection start and a detection end by the shape detecting means,
3. The operation input device according to claim 2, further comprising a sound presenting unit for presenting to the operator by sound when the detection fails and the direction detection by the direction detecting unit occurs. 9. The operation input device according to claim 8, wherein said sound presenting means presents different sounds depending on the type of operation and the type of detected direction. 10. A method for detecting a direction of an object by analyzing a trajectory of position information of the object to be detected, wherein the object moves by a predetermined distance in a predetermined direction and is opposite to the direction. A direction detecting method comprising: detecting a movement in the direction when the object moves by a predetermined distance in a side direction. 11. A method for detecting the direction of an object by analyzing a trajectory of position information of the object to be detected, wherein the object is stationary or within a predetermined time within a predetermined time period or has little motion. After staying in, moving for a predetermined distance in a predetermined direction, moving for a predetermined distance in a direction opposite to the direction, and again staying stationary or with little movement for a predetermined time, moving in the direction A direction detection method, wherein the direction is detected as a motion. 12. The apparatus according to claim 1, wherein the direction detecting means detects the movement in the direction when the object moves by a predetermined distance in a predetermined direction and moves by a predetermined distance in a direction opposite to the direction. The operation input device according to claim 2, wherein: 13. The apparatus according to claim 1, wherein the direction detecting means moves the object in a predetermined direction for a predetermined distance after the object stays in a predetermined range for a predetermined period of time in a stationary state or with a small amount of movement. 10. The operation input device according to claim 2, wherein the movement input in the direction is detected when the movement in the direction is performed by a predetermined distance, and the movement is stopped for a predetermined time again or the movement is small. 14. The sound presenting means presents a first sound when moved in a predetermined direction, moves back to the opposite side of the direction, and returns a second sound when detected as movement in the direction. The operation input device according to claim 9, wherein the sound is presented. 15. The sound presenting means according to claim 14, wherein the first sound presents a different sound depending on the moving direction, and the second sound presents the same sound regardless of the direction. Operation input device. 16. An operation input device according to claim 2, wherein said direction detecting means distinguishes between an intentionally moving operation and an unintentionally occurring operation. 17. The apparatus according to claim 17, wherein the direction detecting means includes means for detecting whether the object has moved in any one of a plurality of predetermined directions, and means for removing a return operation occurring between successive movements in the same direction. The operation input device according to claim 2, further comprising: 18. The direction detecting means has means for detecting a distance of a moving object from a sensor, and a distance in a direction opposite to the direction is long during a continuous movement in a predetermined direction. 18. The operation input device according to claim 17, wherein when the movement is detected, the movement is determined to be a returning movement and is removed. 19. The direction detecting means has means for measuring a time interval between movements of an object, and a movement in a direction opposite to the direction is detected during a continuous movement in a predetermined direction. 18. The operation input device according to claim 17, wherein when the time interval is equal to or less than a predetermined length, the movement during this time is determined to be a return movement and is removed. 20. The operation input device according to claim 17, further comprising sound presenting means for presenting a different sound in the detected direction. 21. The operation input device according to claim 2, wherein the direction detection means detects a trajectory where the start point and the end point of the movement of the object are close to each other. 22. A shape detecting means for detecting a shape of an object to be detected; a menu display means for displaying an operation menu associated with the shape detected by the shape detecting means; An operation input device comprising: a menu selection unit for selecting the operation menu according to the shape detected by the detection unit. 23. The operation input device according to claim 22, wherein the menu selection means presents different shapes when the operation menu is hierarchically traversed. 24. A direction detecting means for detecting a direction of a movement of an object to be detected, a menu displaying means for displaying an operation menu associated with the movement direction detected by the direction detecting means, An operation input device, comprising: a menu selection unit that selects an operation menu different from that detected only once when the movement direction is continuously detected by the direction detection unit. 25. A shape detecting means for detecting a shape of an object to be detected, a direction detecting means for detecting a direction of movement of the object to be detected, and a shape and a movement direction to be detected. An operation input device comprising: menu display means for displaying an operation menu to be displayed; and menu selection means for selecting an operation menu in association with the detected shape and movement direction.