CN103713737A - Virtual keyboard system used for Google glasses - Google Patents

Abstract

The invention discloses a virtual keyboard system for smart glasses. The hand tracking system and the self-positioning system are integrated on the main body of the smart glasses; the hand tracking system is used to detect the three-dimensional positioning information, spatial position and angle of the hand; the self-positioning system is fixedly connected to the On the main body of the smart glasses, it is used to obtain real-time natural motion data of the human head, and calculate the spatial direction, displacement, and speed of the real-time natural motion of the human head; the main body of the smart glasses collects information from the hand tracking system and positioning system, and calculates real-time relative motion data. The present invention can effectively avoid the false response of the interactive system through the associated mapping between the physical plane and the screen; the manual operation on the physical plane can effectively alleviate the fatigue caused by the operation of the suspended arm; it provides an interactive operation interface similar to the traditional one, which can adapt to the operator It is easy to use due to its unique operating habits; the introduction of the self-positioning system can effectively eliminate the problem of operational deviation caused by the movement of the human head.

Description

Virtual keyboard system for smart glasses

technical field

The invention relates to the field of artificial intelligence, in particular to a virtual keyboard system for smart glasses.

Background technique

In recent years, the development of smart wearable devices has become more and more popular. Smart wearable devices include smart watches, smart glasses, smart bracelets and other products. Among them, smart glasses are similar to smartphones and have an independent operating system. Users can install programs provided by software service providers such as software and games, and complete adding schedules, map navigation, interacting with friends, taking photos and other activities through voice or motion control. It is a general term for a class of glasses that have functions such as video and video calls with friends, and can realize wireless network access through mobile communication networks.

Google Glass is an "augmented reality" glasses released by Google in April 2012. It has the same functions as a smartphone, and can be controlled by voice to take pictures, make video calls and identify directions, as well as surf the Internet, process text messages and emails. wait. However, the current way of human-computer interaction with smart glasses is mainly through voice interaction or human eye motion capture, and there is no other better interaction method that allows smart glasses to respond to human command signals.

Due to the limitations of its input signal, voice interaction technology, as a supplement to the existing technology, cannot completely solve the problem of human-computer interaction. And it can only send commands to the computer once and for all through human voice commands. It is impossible to allow people to interact with the computer as quickly and accurately as a mouse, and cannot meet the fast and accurate input requirements of control signals.

Human eye motion capture control is the response of smart glasses by capturing the rotation and blinking of human eyeballs. The control mode is relatively limited, and there are shortcomings such as inaccurate operation and fatigue when using it.

The traditional way of human-computer interaction is to use human body language to send command signals to the computer, such as using fingers to send signals to the computer through the mouse, controlling games through the body, and using fingers to click on the touch screen to send signals, etc. Its development is relatively mature, and the most classic configuration is the combination of keyboard and touchpad of a notebook computer, which can realize convenient and efficient input.

For see-through smart glasses, it is necessary to provide an interactive solution that is accurate, efficient and less likely to cause fatigue, so that smart glasses can work as an independent working unit, which can greatly expand the application range of smart glasses , and can even be used as a portable personal computer.

Using a keyboard layout similar to that of a traditional laptop, and building a simulated virtual keyboard system around smart glasses, it can not only adapt to people's usage habits, but also help realize accurate and efficient interactive operations. As for the problem of fatigue caused by hand-hanging operations, you can use flat objects (such as tables, books, or even walls, etc.) that are ubiquitous in life as carriers, and perform corresponding actions on the plane of such objects with your hands. Map the human hand movements to the virtual keyboard to form a corresponding simulated interactive environment.

In Patent Application No. 201310263439.9, "A Smart Glasses System and Interaction Method for Human-Computer Interaction", an input method is proposed: using a binocular infrared device to perform three-dimensional tracking and positioning of the human hand suspended in front of the eyes to obtain gesture actions as Input, suitable for key operations on the suspended screen simulated in smart glasses.

The above method has the following disadvantages: 1. The human body will have a certain degree of natural swing. For example, the human head is not absolutely still in the natural posture, but has some shaking in real time. The amplitude and direction of this shaking are Unpredictably, when the human hand performs an operation in the air, if the human head shakes in a specific direction at this time, but the human hand does not move, the binocular infrared device may track the "click" action of the hand relative to the head , the binocular device recognizes the click action of the relative movement between the glasses projected screen and the hand, and cannot determine whether it is caused by the movement of the hand or the natural swing of the human head, which may easily cause false triggers and affect the quality of control. High concentration, keeping the head as still as possible to ensure precise input can easily cause fatigue and trance.

2. Since the hand needs to be suspended in the air for three-dimensional operation, it is impossible to avoid the problem of fatigue caused by long-term operation. In the actual experience of personal products, using the three-dimensional gesture operation method similar to the Leap Motion device, usually the arm will produce obvious fatigue after 10 minutes. soreness.

3. It does not provide a universal and precise operation interface, and cannot meet diverse interaction requirements, and can only provide auxiliary operation input.

From the above analysis, the solution proposed in patent application No. 201310263439.9 is suitable for temporary or non-sensitive operating situations, and is not suitable for the smart glasses environment that requires frequent interaction and requires efficient and accurate interaction.

The existing smart glasses on the market usually respond through voice interaction or human eye motion capture, which has the following disadvantages: 1. Voice interaction has limitations, and can only perform intermittent input operations, and cannot perform fast and accurate input; 2. Voice interaction It is prone to fatigue and cannot adapt to long-term interaction requirements; 3. Voice interaction is suitable as a supplementary interaction method, and it is difficult to perform independent pure voice interaction; 4. Human eye motion capture is prone to misoperation, and human eyes have Involuntary movements are generated, which can easily cause false triggers; 5. Human eye motion capture is easy to cause fatigue, which is not conducive to long-term interaction; 6. Human eye motion capture has fewer modes, which is not conducive to complex manipulation .

Contents of the invention

The present invention aims to solve the above-mentioned technical problems existing in the prior art, and provides a virtual keyboard system for smart glasses.

The technical solution of the present invention includes a virtual keyboard system for smart glasses, including a smart glasses main body, a hand tracking system and a self-positioning system, and the hand tracking system and the self-positioning system are integrated on the smart glasses main body; The hand tracking system is used to detect the three-dimensional positioning information, spatial position and angle of the human hand, and transmit it to the main body of the smart glasses; the self-positioning system is fixedly connected to the main body of the smart glasses to obtain the real-time natural movement of the human head Data, calculate the spatial direction, displacement, and speed of the real-time natural movement of the human head, and transmit it to the main body of the smart glasses; the main body of the smart glasses collects information from the hand tracking system and positioning system, and calculates real-time relative motion data.

Preferably, the smart glasses main body includes a smart glasses frame, a display module, and a computing module; the smart glasses frame is used for a hand tracking system and a self-positioning system, and provides a carrier for all modules of the system; the display module is used for providing projection display , and provide a display of a simulated keyboard and a touch panel; the computing module collects the information of the hand tracking system and the positioning system, calculates real-time relative motion data, and obtains effective operation information.

Preferably, the projection display includes a main display area, a virtual keyboard display area and a touchpad display area.

Preferably, the hand tracking system includes a binocular camera module and a plane detection module, the binocular camera module is used to detect the depth of field image of the human hand, obtain the three-dimensional positioning information of the human hand, and transmit it to the main body of the smart glasses; The plane detection module is used to detect the spatial position and angle of the plane where the human hand is located, and transmit it to the main body of the smart glasses.

Preferably, the binocular camera module includes two cameras and a binocular camera processing unit, the binocular camera processing unit collects images from the cameras, and uses a stereo algorithm to calculate the position of the fingertip.

Preferably, the plane detection module is composed of at least three rangefinders with fixed orientations and different orientations.

Preferably, the range finder is a laser range finder or an infrared range finder.

Preferably, the self-positioning system includes a positioning sensor and a positioning unit, the positioning sensor is fixedly connected to the main body of the smart glasses, and is used to obtain real-time natural motion data of the human head; the positioning module obtains the positioning sensor Calculate the spatial direction, displacement, and velocity of the real-time natural movement of the human head, and transmit it to the main body of the smart glasses.

Preferably, the positioning sensor adopts a gyroscope or an accelerometer.

The beneficial effects of the present invention include: through the associated mapping between the physical plane and the screen, the false response of the interactive system can be effectively avoided; the manual operation on the physical plane can effectively alleviate the fatigue caused by the operation of the suspended arm; and an interactive operation interface similar to the traditional one is provided , can adapt to the operator's operating habits and is easy to use; through the introduction of the self-positioning system, the problem of operational deviation caused by the movement of the human head can be effectively eliminated.

Description of drawings

Fig. 1 is a working flow chart of the virtual keyboard system according to the embodiment of the present invention.

Fig. 2 is a schematic diagram of binocular camera stereotaxic positioning of the virtual keyboard system according to the embodiment of the present invention.

Fig. 3 is a schematic diagram of three-dimensional plane positioning of the virtual keyboard system according to the embodiment of the present invention.

FIG. 4 is a schematic diagram of the display output area of the virtual keyboard system according to the embodiment of the present invention.

Detailed ways

The present invention will be described in further detail below in conjunction with the accompanying drawings and specific embodiments.

The invention takes smart glasses as the main body, and implements the functions of the smart glasses virtual keyboard system by integrating a hand tracking system and a self-positioning system.

An embodiment of the present invention provides a virtual keyboard system for smart glasses, as shown in Figure 1, including a smart glasses main body 1, a hand tracking system 2, and a self-positioning system 3, and the operating environment of the virtual keyboard system can use objects commonly used in daily life The physical plane is required to be non-transparent, which can be the plane of tables, flat books, walls, windowsills, stools and similar non-transparent objects.

The hand tracking system 2 and the self-positioning system 3 are integrated on the smart glasses main body 1; the hand tracking system 2 is used to detect the three-dimensional positioning information, spatial position and angle of the hand, and transmit them to the smart glasses main body 1; the self-positioning system 3 is fixedly connected to the smart glasses body. On the glasses main body 1, it is used to obtain the real-time natural movement data of the human head, calculate the spatial direction, displacement, and speed of the real-time natural movement of the human head, and transmit it to the smart glasses main body 1; the smart glasses main body 1 collects the human hand tracking system 2 and positioning system information to calculate real-time relative motion data.

The virtual keyboard system needs to be pre-positioned before use, and the operation of the human hand on the selected physical plane is mapped to the projected virtual keyboard to generate corresponding interactive effects.

In the embodiment of the present invention, the false response of the interactive system can be effectively avoided through the associated mapping between the physical plane and the screen; the manual operation on the physical plane can effectively alleviate the fatigue caused by the operation of the suspended arm; providing an interactive operation interface similar to the traditional one can It adapts to the operator's operating habits and is easy to use; through the introduction of the self-positioning system 3, the problem of operational deviation caused by the movement of the human head can be effectively eliminated.

The smart glasses main body 1 includes a smart glasses frame, a display module, and a computing module; the smart glasses frame uses the basic frame of smart glasses, which is used for the hand tracking system 2 and the self-positioning system 3, and provides the carrier for all modules of the system;

The smart glasses frame includes the physical frame main body of commonly used smart glasses, a microprocessor, an independent power supply, a wireless connection device and a projection display device.

The display module is used to provide projection display, and to provide the display of the simulated keyboard and touch panel; the calculation module collects the information of the hand tracking system 2 and the positioning system, calculates real-time relative motion data, and obtains effective operation information.

The projected display includes a main display area, a virtual keyboard display area and a touchpad display area. It displays the virtual human hand above the interactive area according to the position of the actual human hand.

Preferably, the hand tracking system 2 includes a binocular camera module and a plane detection module that provide 3D positioning information of the hand relative to the smart glasses, the binocular camera module is used to detect the depth of field image of the hand, obtain the three-dimensional positioning information of the hand, and send it to The smart glasses main body 1; the plane detection module is used to detect the spatial position and angle of the plane on which the human hand is located, and transmit the information to the smart glasses main body 1.

The binocular camera module is used for spatial positioning and tracking of the operator's fingertips when the virtual keyboard system is working. It includes two cameras and a binocular camera processing unit. The binocular camera processing unit collects images from the cameras and calculates them using a stereo algorithm. The location of the fingertips.

According to the needs of the virtual keyboard system, the binocular camera module realizes real-time positioning and tracking of the quality inspection of the fingers of both hands. The two cameras of the module use the stereo algorithm to calculate the real-time Position, the basic principle is shown in Figure 2. When the tracking determines that the fingertip is in contact with the physical plane used at a certain position, the effect of the virtual keyboard system is that the finger is pressed at the corresponding virtual position, and a corresponding input response is generated. .

Parallax d=x _l -x _r =fb/z,

Among them, f is the focal distance, b is the baseline, and z is the longitudinal distance.

Since the system can track ten fingers at the same time, it can support 10 fingers to press at the same time for input. When it is determined that the human hand has moved to the position of the touchpad, similarly, operations such as touching, sliding, and double-clicking on the desktop by the human hand are equivalent to performing the same operation on the virtual touchpad under the virtual keyboard.

The plane detection module provides 3D positioning information of the physical plane used relative to the smart glasses. It is composed of at least three rangefinders with different fixed orientations. The three points positioned by the above rangefinders are used to locate the physical plane used in the operating environment. Spatial location.

Preferably, the range finder is a laser range finder or an infrared range finder.

The plane detection module provides the position and orientation information of the physical plane used relative to the smart glasses. The light emitted by each rangefinder will generate an intersection with the physical plane, and three intersections will generate three intersections. According to the data of the rangefinder, it can The distance data of the three intersection points relative to the rangefinder group are located in space, and the direction and distance of each point are determined. The spatial position of the point relative to the rangefinder group can be determined. According to the data of the three points The relative spatial position of the physical plane relative to the rangefinder group can be determined, as shown in FIG. 3 , and the 3D positioning information of the physical plane relative to the smart glasses can be obtained.

The self-positioning system 3 is to detect the disturbance of the binocular camera due to the natural swing of the human head in real time when the virtual keyboard system is working, and to avoid the deviation of the system positioning caused by the movement of the human head, including a positioning sensor and a positioning unit .

The positioning sensor is fixedly connected to the main body 1 of the smart glasses, and is used to obtain real-time natural motion data of the human head; the positioning module obtains the data of the positioning sensor, calculates the spatial direction, displacement, and speed of the real-time natural motion of the human head, and transmits the data to the Smart glasses main body 1.

Preferably, the positioning sensor adopts a gyroscope or an accelerometer.

The working process of the embodiment of the present invention:

1. Before the virtual keyboard system starts working, first select a flat, non-transparent physical plane to avoid the disorder of the tracking system caused by light refraction, and place your hands on the plane naturally in the posture of operating a traditional keyboard;

2. After starting the virtual keyboard system, the system sets the pre-positioning, so as to provide hand, plane and corresponding initial positioning. The completion of this process includes the positioning of the human hand relative to the smart glasses and the positioning of the physical plane used relative to the smart glasses.

3. After the initial positioning is completed, the system can work normally. The simulated hand in the virtual keyboard system will move in strict accordance with the position of the human hand. Click action, also can move the hand to the touch operation area, can simulate the control of the touchpad, as shown in Figure 4. Responding strictly according to the movement of the human hand is aimed at the natural swing of the human head, and the swing of the head during the interactive operation will not affect the movement of the simulated hand in the simulated image. The calculation module obtains the head movement information in real time according to the self-positioning system 3, and deducts the real-time movement data of the head from the real-time relative movement data of the head and hands to obtain the effective operation information of the hand, thereby realizing precise interactive operation.

Embodiments of the present invention have the following effects:

①. Through the association mapping between the physical plane and the screen, the false response of the interactive system can be effectively avoided;

②. Human hands operate on a physical plane, which can effectively relieve the fatigue caused by the operation of suspended arms;

③. Provide an interactive operation interface similar to the traditional one, which can adapt to the operator's operating habits and is easy to use;

④. Through the introduction of the glasses self-positioning system, the problem of operation deviation caused by the movement of the human head can be effectively eliminated;

⑤. Smart glasses can be conveniently used as an independent terminal with complete input functions anytime and anywhere, which is conducive to the popularization and application of smart glasses.

The specific embodiments of the present invention described above do not constitute a limitation to the protection scope of the present invention. Any other corresponding changes and modifications made according to the technical concept of the present invention shall be included in the protection scope of the claims of the present invention.

Claims (9)

1. A virtual keyboard system for smart glasses, characterized in that it includes a smart glasses main body, a hand tracking system and a self-positioning system, and the hand tracking system and the self-positioning system are integrated on the smart glasses main body; The hand tracking system is used to detect the three-dimensional positioning information, spatial position and angle of the human hand, and transmit it to the main body of the smart glasses; the self-positioning system is fixedly connected to the main body of the smart glasses to obtain the real-time natural movement of the human head Data, calculate the spatial direction, displacement, and speed of the real-time natural movement of the human head, and transmit it to the main body of the smart glasses; the main body of the smart glasses collects information from the hand tracking system and positioning system, and calculates real-time relative motion data. 2. The virtual keyboard system for smart glasses as claimed in claim 1, wherein the smart glasses main body includes a smart glasses frame, a display module, and a computing module; the smart glasses frame is used for hand tracking systems and automatic The positioning system provides the carrier of all modules of the system; the display module is used to provide projection display, and provides the display of the simulated keyboard and touch panel; the operation module collects the information of the human hand tracking system and the positioning system, and calculates real-time relative motion Data to obtain effective operation information. 3. The virtual keyboard system for smart glasses according to claim 2, wherein the projected display comprises a main display area, a virtual keyboard display area and a touchpad display area. 4. The virtual keyboard system for smart glasses according to claim 1, wherein the hand tracking system includes a binocular camera module and a plane detection module, and the binocular camera module is used to detect the depth of field image of the human hand , obtain the three-dimensional positioning information of the human hand, and transmit it to the main body of the smart glasses; the plane detection module is used to detect the spatial position and angle of the plane where the human hand is located, and transmit it to the main body of the smart glasses. 5. The virtual keyboard system for smart glasses as claimed in claim 4, wherein the binocular camera module includes two cameras and a binocular camera processing unit, and the binocular camera processing unit collects the camera image, using a stereo algorithm to calculate the position of the fingertip. 6 . The virtual keyboard system for smart glasses according to claim 4 , wherein the plane detection module is composed of at least three range finders with different fixed orientations. 7 . 7. The virtual keyboard system for smart glasses according to claim 6, wherein the range finder is a laser range finder or an infrared range finder. 8. The virtual keyboard system for smart glasses according to claim 1, wherein the self-positioning system includes a positioning sensor and a positioning unit, and the positioning sensor is fixedly connected to the main body of the smart glasses for Obtain real-time natural motion data of the human head; the positioning module obtains the data of the positioning sensor, calculates the spatial direction, displacement, and speed of the real-time natural motion of the human head, and transmits the data to the main body of the smart glasses. 9. The virtual keyboard system for smart glasses according to claim 8, wherein the positioning sensor adopts a gyroscope or an accelerometer.

Images