JP2010184600A - Onboard gesture switch device - Google Patents

Abstract

There is provided a gesture switch device that can be operated easily and reliably with a feeling that a switch operation unit displayed on an image display device installed in a vehicle is actually touched and operated.
An image display device that displays a switch operation unit, a camera that captures a driver's hand movement toward the image display device, and image information of the hand movement is input from the camera, and based on the input information. And a computer that displays a response indicating that the switch operation unit is operated when it is determined that the operation is performed on the switch operation unit. A virtual operation of a hand indicating an operation on the switch operation unit is performed within a designated area in the space between the sensors, and a sensor for detecting a start operation and an end operation of the operation is provided. Let the camera start shooting and end shooting.
[Selection] Figure 2

Description

  The present invention relates to a vehicle-mounted gesture switch device, and more specifically, can be easily and reliably operated as if the user actually touched and operated a switch operation unit of an image display device provided in a vehicle.

  Cars are equipped with various devices such as navigation, audio equipment, and air conditioning equipment, and the number of switches that control these equipment is also increasing. In recent years, an image display device equipped with a touch panel type switch operation unit has been installed, and the switch operation can be performed while switching the display screen, so that the driver or passenger can control each device while driving or temporarily stopping. There are many things to do.

  However, by staring at the image display device for switch operation or the like, the driver's line-of-sight direction may deviate from the original line-of-sight direction, which may hinder safe driving. In order to enhance safety, it is preferable that the image display device is installed at a deeper position on the instrument panel, and the viewing angle when viewing the image display device is made as small as possible. However, when the image display device is installed at the above position, the distance from the driver's seat to the image display device is widened, which makes it difficult for the driver to directly touch and operate the switch operation unit displayed as an image.

To cope with this problem, a composite switch that integrates a plurality of switch functions is provided at a position where the driver can reach, and the switch displayed on the image display device is operated using the composite switch. It is possible to do. For example, Japanese Patent Laid-Open No. 10-112236 (Patent Document 1) discloses a navigation switch direction instruction, audio device selection, and air-conditioning device operation mode by a composite switch combining a multipoint tilt type selection switch and a volume. Selection, expansion and reduction of the navigation screen, volume control, temperature control, etc. are possible.
However, the composite switch is not easy to operate, and if the switch of the image display and the composite switch to be actually operated are separated from each other, it is more difficult to operate, and erroneous operation is likely to occur. That is, the above-described configuration has a problem that an intuitively understandable operation such as touching and operating the switch operation unit displayed as an image is not possible.

JP-A-10-112236

  The present invention has been made in view of the above problems, and it is an object of the present invention to be able to operate easily and reliably with a feeling of actually touching and operating a switch operation unit displayed on an image display device installed in a vehicle. Yes.

In order to solve the above problems, the present invention provides an image display device for displaying a switch operation unit mounted in front of a driver's seat of a vehicle and in a visual field region in a driving state of the driver;
A camera for photographing the movement of the driver's hand toward the image display device;
The driver's hand movement image information is input from the camera, and based on the input information, image processing is performed to determine whether the operation is for the switch operation unit displayed on the image display device, and the switch operation unit And a computer that displays a response indicating that the switch operation unit displayed as an image is operated when it is determined that the operation is
The movements of various switch operation hands for each switch operation unit displayed on the image display device are virtual operations designated in advance, and within a designated area specified in the space between the image display device and the driver's seat. The virtual motion of the designated hand shall be performed,
A start operation indicating the start of the operation on the switch operation unit displayed on the image and an end operation indicating the end are specified, a sensor for detecting the start operation and the end operation is installed, and a detection signal of the sensor is used. The camera starts shooting, ends shooting, and
The response display indicating that the switch operation unit is operated provides an in-vehicle gesture switch device that displays an image in a state where the corresponding switch operation unit is operated and / or displays an operation content as a voice display. .

  According to the above configuration, since the computer performs image processing to determine whether or not the movement of the driver's hand photographed by the camera is an operation on the switch operation unit displayed on the image display device, the driver The operation can be completed only by performing a virtual operation (gesture) for operating the switch operation unit of the image display device without requiring a complicated operation such as a composite switch. In addition, by enabling such virtual operation, it is possible to reduce the number of hard switches and improve the design.

  At this time, as described above, the start operation and the end operation for starting the operation on the switch operation unit displayed as an image are specified, and the shooting by the camera is not started unless the start operation is detected by the sensor. It is possible to prevent other operations that are not switch operations from being shot and erroneously determined as switch operations. As a sensor for detecting the start operation and the end operation, for example, a photo sensor or an ultrasonic sensor is preferably provided at a position where it can be detected. As a result, as a start operation and an end operation, the sensor sends a detection signal to the computer and starts or ends the shooting by the camera only by performing a predetermined operation such as holding a hand toward the sensor for a certain period of time. it can.

In addition, as described above, by limiting the area where the virtual motion of the hand is performed to the specified area specified in the space between the image display device and the driver's seat, the area is set as a determination area for the switch operation. It is possible to prevent erroneous determination and malfunction by using an image of an irrelevant portion outside the designated area.
Further, as described above, when the computer determines that the operation is performed on the switch operation unit, the driver displays the response display with an image or sound indicating that the predetermined switch operation unit displayed on the image is operated. Can easily confirm whether or not an accurate switch operation has been performed, and can improve operability.

  The virtual movement of the hand specified in advance is similar to the movement of the hand at the time of actual switch operation, the operation of pressing the button on the switch operation part is the operation of tilting the finger of the hand forward, and the operation of turning the rotary dial is rotation It is preferable that the operation of rotating the hand in a finger posture such as grasping the dial and the operation of sliding the slide switch are operations of sliding the hand in a finger posture so as to grasp the slide switch.

  As described above, the virtual operation of the hand imitates the movement of the hand at the time of actual switch operation, so that the troublesomeness of storing a special virtual operation in advance is eliminated, and the switch operation unit displayed on the image display device Intuitive switch operation is possible with the feeling of actually touching and operating.

Specifically, for example, an image of a finger posture with a finger (for example, an index finger), an image of a finger posture holding a rotating dial, and an image of a finger posture holding a slide switch Is stored in the memory of the computer as a standard pattern for button operation, rotary dial operation, and slide switch operation, and the computer performs button matching by performing pattern matching between the hand image captured by the camera and the standard pattern. It is possible to determine which of the rotary dial operation and the slide switch operation is being performed. As for the button operation, it can be determined which button is selected by the position coordinates of the fingertip. Furthermore, by tracking the fingertip position coordinates, it detects the displacement of the finger, detects the movement of tilting the finger forward (pressing the button), the rotation direction and angle of the rotary dial, the slide direction and slide of the slide switch The amount can also be detected. That is, the operation amount of each switch operation can also be determined.
Based on these image processing results, the computer can perform response display (image display or / and audio display) indicating that a predetermined switch operation unit has been operated, and can appropriately control the device to be operated. .

  The virtual operation of the hand designated in advance for the navigation switch operation unit may be an operation of enlarging the display screen by moving the hand toward the image display device or an operation of reducing the display screen by moving the hand away. preferable.

  According to the above-described configuration, for example, a fingertip is applied to the switch operation unit (hereinafter referred to as an enlargement / reduction switch) for enlarging or reducing the screen displayed on the navigation display screen indicating map information or the like. The user can zoom in and out by simply moving his / her hand close to or away from the image display device while holding the finger in a gripping posture. Can be reduced.

Specifically, an image of a predetermined finger posture indicating the operation of the enlargement / reduction switch on the display screen (for example, an image of a finger posture where the fingertip is placed on the enlargement / reduction switch, or a finger posture such as holding the switch Is stored in the memory of the computer as a standard pattern for enlargement / reduction operation, and the computer performs the enlargement / reduction operation by performing pattern matching between the image captured by the camera and the standard pattern. A determination of whether or not can be made. Furthermore, by tracking the position coordinates of the finger, it is possible to detect the displacement of the finger, determine whether to enlarge or reduce the display screen according to the displacement direction, and also determine the magnification of enlargement or reduction from the amount of displacement. it can.
In addition, an operation function for scrolling the screen is added to the enlargement / reduction switch. If the direction of finger displacement is not in the front-rear direction but in the up / down / left / right direction, it is determined that the display screen is scrolled, and the display screen such as a map is displayed. It is also possible to scroll in the vertical and horizontal directions.

  It is preferable that a stereo camera is provided as the camera, and the hand movement is input from the stereo camera to the computer as three-dimensional information.

The stereo camera is a camera that can reproduce binocular parallax by arranging two lenses in parallel at a predetermined interval, and can simultaneously shoot an object to be photographed from different directions. As described above, by capturing the hand movement with the stereo camera, the shape, distance, and position of the captured hand can be recognized in three dimensions. In particular, a stereo camera can clearly capture a three-dimensional movement that simulates an actual hand movement during switch operation. Therefore, it is possible to accurately recognize the shape of the photographed hand, measure the distance, and the like, and to further increase the accuracy of determining the switch operation.
In addition, since the switch operation determination area can be clearly set by the three-dimensional coordinates, it is possible to effectively prevent an irrelevant portion of the image from being used for the switch operation determination and erroneous determination or malfunction.
The installation position of the stereo camera is not particularly limited as long as the driver can shoot a virtual motion performed toward the switch operation unit of the image display device, but for example, an image attached near the upper center of the instrument panel. It is preferable to be provided in the lower part of the display device.

  As described above, according to the present invention, the computer performs image processing to determine whether the movement of the driver's hand taken by the camera is an operation on the switch operation unit displayed on the image display device. Therefore, the driver can complete the operation only by performing a virtual operation (gesture) such as operating the switch operation unit of the image display device. That is, it is possible to simplify the switch operation and reduce the operation burden on the driver. In addition, by enabling such virtual operation, it is possible to reduce the number of hard switches and improve the design.

  In addition, as described above, by specifying the start operation and the end operation for starting the operation on the switch operation unit displayed as an image, and by not starting the shooting by the camera unless the start operation is detected by the sensor, It is possible to prevent another operation of the driver that is not a switch operation from being shot and erroneously determined as a switch operation. Further, as described above, the region in which the virtual motion of the hand is performed is limited to the specified region specified in the space between the image display device and the driver's seat, and the region is used as the determination region for the switch operation. The image of the irrelevant portion outside the designated area is used for the determination, and it is possible to prevent erroneous determination and malfunction. That is, with the above configuration, the reliability of the switch operation can be improved.

  Further, as described above, when the computer determines that the operation is performed on the switch operation unit, the driver displays the response display with an image or sound indicating that the predetermined switch operation unit displayed on the image is operated. Can easily confirm whether or not an accurate switch operation has been performed, and can improve operability.

  In addition, by shooting with a stereo camera, it is possible to clearly capture the three-dimensional hand movement that is actually performed by touching the switch operation unit of the image display device. Therefore, it is possible to accurately recognize the shape of the photographed hand, measure the distance, and the like, and to increase the determination accuracy of the switch operation.

It is the schematic which shows the installation state of the vehicle-mounted gesture switch apparatus in 1st Embodiment of this invention. It is the schematic which shows the state which is performing virtual operation. It is drawing which shows the display screen of an image display apparatus. It is a figure which shows the structure of a gesture switch apparatus. It is a flowchart of gesture switch operation. (A) is an initial screen for response display, and (B) is a final screen for response display. The other virtual motions are shown, (A) is a virtual motion of an operation of pressing a button, and (B) is a virtual motion of an operation of sliding a slide switch. The display screen in 2nd Embodiment is shown, (A) is the display screen before expansion operation, (B) is the display screen after expansion operation. The display screen in 2nd Embodiment is shown, (A) is the display screen before a scroll, (B) is the display screen after a scroll.

Embodiments of the present invention will be described with reference to the drawings.
1st Embodiment is shown in FIG. 1 thru | or FIG.
The in-vehicle gesture switch device 10 according to the present embodiment includes an image display device 11 that displays a switch operation unit 13, a stereo camera 14 that captures the movement of a driver's hand toward the image display device 11, and the stereo camera 14. The computer 15 that determines whether or not the operation is performed on the switch operation unit 13 by performing image processing on the input 3D image information of the hand movement, and a start operation and an end operation that start the operation on the switch operation unit 13 are detected. A sensor 16 is provided. Further, the computer 15 is connected to navigation, audio equipment, and air conditioning equipment, which are switch operation target equipment 19.

  The image display device 11 is provided in front of the driver's seat 18 and in a visual field region in the driving state of the driver. In the present embodiment, the image display device 11 is provided at a slightly recessed position near the upper center of the instrument panel 17. . As shown in FIG. 3, the display screen 12 of the image display apparatus 11 includes a navigation operation button 13a, an audio device operation button 13b, an air conditioner operation button 13c, and a volume operation button as a switch operation unit 13. A rotary dial 13d for the purpose and a slide switch 13e for enlarging / reducing the navigation display screen are displayed.

  In the in-vehicle gesture switch device 10 according to the present embodiment, each switch operation unit 13 displayed on the display screen 12 of the image display device 11 can be directly touched and operated, but hand movement during actual operation is possible. The switch operation unit 13 can be operated also by the driver performing a virtual operation that simulates the above.

  The stereo camera 14 captures a driver's hand movement (virtual motion) in three dimensions, and outputs the three-dimensional image information obtained by the capture to a computer 15 connected to the stereo camera 14. No. 14 is installed at a position where the movement of the driver's hand can be reliably photographed (in the present embodiment, the lower part of the image display device 11).

  The sensor 16 detects a start operation for starting a switch operation based on a virtual operation of the driver and an end operation for ending the switch operation. The sensor 16 is connected to the sensor 16 when the start operation or the end operation is detected. A detection signal is output to the computer 15. The computer 15 outputs a command signal that causes the stereo camera 14 to start or stop shooting based on the detection signal from the sensor 16, and the stereo camera 14 starts or ends shooting based on the signal. The sensor 16 is preferably installed at a position where a start operation and an end operation by the driver can be easily detected. In the present embodiment, a photo sensor is provided as a sensor 16 near the lower center of the instrument panel 17. Yes. The start operation and end operation are specified in advance according to the form of the sensor 16, and in the present embodiment, an operation of holding the hand for a certain time toward the photo sensor 16 is referred to as the start operation and end operation.

As described above, the computer 15 provided in the instrument panel 17 issues an instruction to start and end shooting to the stereo camera 14 based on the detection signal from the sensor 16 and is input from the stereo camera 14. The three-dimensional image information of the virtual operation is subjected to image processing to determine whether or not the switch operation unit 13 is operated. In the memory of the computer 15, an image of a finger posture that holds the index finger up and presses a button, an image of a finger posture that holds the rotary dial, and an image of a finger posture that holds the slide switch By storing in advance as standard patterns of the pressing operation of the buttons 13a, 13b, 13c, the rotating operation of the rotary dial 13d, and the sliding operation of the slide switch 13e, and by performing pattern matching of the input image using the respective standard patterns, The type of operation and the operation amount are determined.
When it is determined that the operation is for the predetermined switch operation unit 13, a command signal for displaying an image while the predetermined switch operation unit 13 is operated is output to the image display device 11 as a response display, and the computer The operation signal corresponding to the switch operation is output to the device 19 to be operated connected to 15. In addition, instead of the image display as the response display, or in addition to the image display, the operation content may be displayed by voice.

In the gesture switch device 10 according to the present invention, a region in which the driver performs a virtual operation for a switch operation, shoots and images the virtual operation, and determines the switch operation is determined in advance.
In the present embodiment, as shown in FIG. 2, a three-dimensional space A having a certain size in the space between the image display device 11 and the driver's seat 18 is set as a designated region for determining the switch operation. Note that the position and size of the designated area A can be appropriately set in consideration of the position, shape, size, displacement, and the like of the hand to be imaged.

Hereinafter, a switch operation using the gesture switch device 10 having the above-described configuration will be described. Here, the rotary dial 13d displayed on the display screen 12 of the image display device 11 is rotated clockwise to increase the volume.
First, a start operation indicating that an operation on the switch operation unit 13 is started is performed (S1). In the present embodiment, an operation of holding the hand over the sensor 16 for a certain time is performed. The stereo camera 14 starts photographing in response to a detection signal from the sensor 16 that has detected the start operation (S2). Subsequently, a virtual operation is performed in which the driver rotates his / her hand in such a posture that the driver grasps the rotary dial 13d within the designated area A (S3, see FIG. 2).

  The computer 15 performs image processing on the three-dimensional image captured by the stereo camera 14. That is, after the captured three-dimensional image is binarized (S4), pattern matching of the binarized image is performed using the standard pattern stored in the memory (S5), and the type of switch operation (rotating dial 13d) Is determined (S6). As a response display, the type of switch operation is displayed as an image, such as turning on the rotary dial 13d displayed on the display screen 12 of the image display device 11 [see S7, FIG. 6A]. Further, the displacement of the finger is detected by tracking the position coordinates of the finger (S8), and the rotation direction and angle of the rotation dial 13d are calculated to determine the operation amount of the rotation operation (S9). As a response display, an operation amount is displayed as an image, for example, the rotation dial 13d displayed on the display screen 12 of the image display device 11 is rotated in accordance with the operation amount [S10, see FIG. An operation signal is output to the controller of the target audio device 19 (S11). When the switch operation is ended, an end operation is performed (S12), and the photographing of the stereo camera 14 is ended (S13). When another switch operation is further performed, the process returns to the step of performing the virtual operation (S3). To continue processing.

  According to the similar steps, the operation buttons 13a, 13b, and 13c can be pressed by a virtual operation of tilting the finger of the hand forward as shown in FIG. Further, as shown in FIG. 7B, the slide operation of the slide switch 13e can be performed by a virtual operation of sliding the hand in a finger posture such that the slide switch 13e is grasped.

  As described above, the computer 15 performs image processing to determine whether or not the movement of the driver's hand photographed by the stereo camera 14 is an operation on the switch operation unit 13 displayed on the image display device 11. The driver can complete the operation only by performing a virtual operation (gesture) such as operating the switch operation unit 13 of the image display device 11. That is, it is possible to simplify the switch operation and reduce the operation burden on the driver. In addition, by enabling such virtual operation, it is possible to reduce the number of hard switches and improve the design.

  Further, as described above, the start operation and the end operation for starting the operation on the switch operation unit 13 displayed as an image are specified, and the photographing by the camera 14 is not started unless the sensor 16 detects the start operation. Therefore, it is possible to prevent another operation of the driver that is not a switch operation from being shot and erroneously determined as a switch operation. Furthermore, as described above, the region in which the virtual motion of the hand is performed is limited to the designated region A specified in the space between the image display device 11 and the driver's seat 18, and the region A is determined as a switch operation determination region. By doing so, an image of an irrelevant portion outside the designated area is used for determination, and erroneous determination and malfunction can be prevented. That is, with the above configuration, the reliability of the switch operation can be improved.

  Further, as described above, when the computer 15 determines that the operation is performed on the switch operation unit 13d, a response display with an image or sound indicating that the predetermined switch operation unit 13d displayed as an image is operated is performed. The driver can easily confirm whether or not an accurate switch operation has been performed, and can improve operability.

  Further, by photographing with the stereo camera 14, it is possible to clearly capture the three-dimensional hand movement that is actually performed by touching the switch operation unit 13 of the image display device. Therefore, it is possible to accurately recognize the shape of the photographed hand, measure the distance, and the like, and to increase the determination accuracy of the switch operation.

8 and 9 show a second embodiment.
The second embodiment is the same as the first embodiment except that the knob-shaped switch 13f that can be enlarged, reduced, and scrolled is used.
The display screen 12 is enlarged by performing a virtual operation in which the hand is brought close to the image display device 11 while holding the finger-like switch 13f in a finger posture [FIG. 8 (A) → (B)], and the hand is moved to the image display device. The display screen 12 is set to be reduced by performing a virtual operation away from 11 [FIG. 8 (B) → (A)]. Further, the display screen 12 is scrolled in the sliding direction by performing a virtual operation of sliding the hand in a plane substantially parallel to the display screen 12 of the image display device 11 with the finger posture holding the switch 13f. It is set. That is, when the hand is slid leftward in a plane substantially parallel to the display screen 12 from the state of FIG. 9A, a display screen as shown in FIG. 9B is obtained.

  Specifically, an image of a finger posture as if holding the knob-shaped switch 13f is stored in the memory of the computer 15 as a standard pattern for enlargement / reduction / scroll operation, and the computer 15 was photographed by the stereo camera 14. By performing pattern matching between the hand image and the standard pattern, it is possible to determine whether or not the operation is an enlargement / reduction / scroll operation. Furthermore, the finger displacement can be detected by tracking the finger position coordinates. If the displacement direction is the forward direction, the display screen 12 is enlarged, and if the displacement direction is the backward direction, the display screen 12 is reduced, and substantially parallel to the display screen. In the vertical and horizontal directions within a simple plane, it can be determined that the display screen is being scrolled, and the enlargement / reduction magnification and scroll amount can also be determined based on the amount of displacement.

  According to the above-described configuration, the hand approaches or moves away from the image display device while holding the finger posture such as grasping the knob-shaped switch 13f displayed on the navigation display screen 12 showing map information or the like. Since the display screen can be enlarged, reduced, and scrolled simply by sliding in a substantially parallel plane, the driver can perform intuitive and easy-to-understand operations, and the operation burden can be reduced.

DESCRIPTION OF SYMBOLS 10 Vehicle-mounted gesture switch apparatus 11 Image display apparatus 12 Display screen 13 Switch operation part 13a, 13b, 13c Button 13d Rotation dial 13e Slide switch 13f Knob-shaped switch 14 Stereo camera 15 Computer 16 Sensor 17 Instrument panel 18 Driver's seat 19 Equipment

Claims (4)

An image display device for displaying a switch operation unit mounted in front of the driver's seat of the vehicle and in a visual field region in the driving state of the driver;
A camera for photographing the movement of the driver's hand toward the image display device;
The driver's hand movement image information is input from the camera, and based on the input information, image processing is performed to determine whether the operation is for the switch operation unit displayed on the image display device, and the switch operation unit A computer for displaying a response indicating that the switch operation unit displayed on the image is operated when it is determined that
The movements of various switch operation hands for each switch operation unit displayed on the image display device are virtual operations designated in advance, and within a designated area specified in the space between the image display device and the driver's seat. The virtual motion of the designated hand shall be performed,
A sensor that detects a start operation and an end operation is specified by specifying a start operation that indicates the start of an operation on the switch operation unit displayed on the image and an end operation that indicates the end, and a detection signal of the sensor The camera starts shooting, ends shooting, and
The response display indicating that the switch operation unit has been operated is an in-vehicle gesture switch device in which an image is displayed in a state where the corresponding switch operation unit is operated and / or operation content is displayed as a voice.   The virtual motion of the hand specified in advance is similar to the movement of the hand during actual switch operation, the operation of pressing the button of the switch operation unit is the operation of tilting the finger of the hand forward, and the operation of turning the rotary dial is the above The in-vehicle gesture switch device according to claim 1, wherein the operation of rotating the hand in a finger posture such as grasping the rotary dial and the operation of sliding the slide switch are operations of sliding the hand in a finger posture such as grasping the slide switch. .   The virtual operation of the hand designated in advance for the navigation switch operation unit is an operation of enlarging the display screen by moving the hand closer to the image display device, or an operation of reducing the display screen by moving the hand away from the image display device. The in-vehicle gesture switch device according to claim 2.   The in-vehicle gesture switch device according to any one of claims 1 to 3, wherein a stereo camera is provided as the camera, and the movement of the hand is input to the computer as three-dimensional information from the stereo camera.

Images