CN104679270B - System and method for receiving user input, program storage medium and program therefor - Google Patents

Abstract

A system and method for receiving user input, a program storage medium and a program thereof. The method for receiving the user input comprises the following steps: displaying a virtual keyboard configuration and a control area, wherein the virtual keyboard configuration comprises a plurality of subgroup keys, and each subgroup key corresponds to one of a plurality of areas of the control area; extracting a plurality of positions of an object from at least one extracted image, and identifying the positions of characteristic points of the object; determining a selection area, wherein the position of the characteristic point in all the areas in the control area is the selection area; determining a plurality of keys corresponding to the selected area; and interpreting the movement of the object as input data to a user interface system.

Description

System and method for receiving user input and its program storage medium and program

technical field

The invention relates to a system and method for receiving user input methods, a program storage medium and a program.

Background technique

In recent years, various user interface systems and systems for receiving user input have been proposed. In general, a user input system refers to a system that interacts with other people via a machine such as a personal computer (PC). The user interface system provides an input method to allow users to operate the system, and also provides an output method to allow users to see the operation results.

What everyone hopes is to provide a simple, efficient and easy-to-use system for receiving user input.

Contents of the invention

The present disclosure relates to a system and method for receiving user input, a program storage medium and a program.

According to an embodiment of the present disclosure, the method for receiving user input includes the following steps: displaying a virtual keyboard configuration and a control area, the virtual keyboard configuration includes a plurality of subgroup keys, and each subgroup key corresponds to one of the plurality of areas of the control area One; extract multiple positions of an object from at least one extracted image, identify the position of the feature point of the object; determine the selected area, and among all the areas in the control area, the position of the feature point is the selected area ; determining a plurality of keys corresponding to the selected area; and interpreting the movement of the object as input data to the user interface system.

Another embodiment of the present disclosure provides a system for receiving user input, the system includes: a display for displaying a virtual keyboard configuration and a control area, the virtual keyboard configuration includes a plurality of subgroup keys, and each subgroup key corresponds to the one of a plurality of areas of the control area; one or more sensors sensing movement of the object; a computing system coupled to the one or more sensors and the display, the computing system extracting multiple images of the object from at least one extracted image position, to identify the position of the feature point of the object, determine the selected area, among these areas in the control area, the position where the feature point is located is the selected area, determine a plurality of keys corresponding to the selected area, and The movement of the object is interpreted as input data to the system.

Yet another embodiment of the present disclosure provides a program storage medium, the stored computer program causes the electronic device to perform the following steps: display a virtual keyboard configuration and a control area, the virtual keyboard configuration includes a plurality of subgroup keys, and each subgroup key is respectively Corresponding to one of the plurality of areas of the control area; extracting multiple positions of the object from at least one extracted image, identifying the position of the feature point of the object; determining the selected area, in the plurality of areas of the control area, the feature The location of the point is the selection area; determining a plurality of keys corresponding to the selection area; and interpreting the movement of the object as input data to the user interface system.

A further embodiment of the present disclosure provides a computer program product stored on a computer-readable medium, including a computer-readable program, for executing the above-mentioned method on an electronic device.

In order to have a better understanding of the above and other aspects of the present application, the following specific embodiments are described in detail as follows in conjunction with the accompanying drawings:

Description of drawings

FIG. 1 illustrates a user interface system disclosed according to an embodiment.

FIG. 2A illustrates a wearable computing device according to an embodiment of a user interface system.

FIG. 2B illustrates an embodiment of operating the user interface system of FIG. 2A .

FIG. 2C illustrates the embodiment of the user interface system of FIG. 2A, showing a mode of virtual input therein.

Figures 3A-3D show an embodiment of the system inputting data at the user level, where the user wears a marker or glove to input data.

Figure 4 is a keypad configuration disclosing another embodiment where the user wears a marker or glove to enter data.

FIG. 5 is another embodiment of the present disclosure, where the user inputs data with bare hands.

FIG. 6 is another embodiment of the present disclosure, the user inputs data with both hands.

FIG. 7 is another embodiment of the present disclosure, illustrating that the control area can be combined with subgroup keys, and the correspondence of the keys can be changed with the movement of the hand.

FIG. 8A and FIG. 8B are another embodiment of the present disclosure, the control area can be combined with subgroup keys, and the correspondence of the keys can be changed with the movement of the hand.

FIG. 8C is another embodiment of the present disclosure, the user types with one finger.

8D to 8E are another embodiment of the present disclosure, the user uses one finger to type.

FIG. 9 shows that a user input system includes a user interface system and a host system according to another embodiment of the present disclosure.

FIG. 10 is the system embodiment of FIG. 9 .

FIG. 11 illustrates a method of an embodiment of the present disclosure applicable to the system shown in FIG. 1 or FIG. 9 .

Fig. 12 illustrates a system for receiving user input according to yet another embodiment of the present disclosure.

In the following detailed description, for purposes of explanation, set forth are specific details in order to provide a comprehensive understanding of the disclosed embodiments. These embodiments can be practiced even if specific details are not expressly stated. In other instances, well-known structures and devices in the drawings are simplified and schematically disclosed.

【Symbol Description】

100, 200, 900A: user interface system

110, 208, 902: Sensors

120, 904, 914: Processor

130: memory

140, 202, 918: display

204: Computer Systems

206: Camera

240: Virtual input pattern

241: QWERTY-like keyboard

242: Control area

310: hand

320: Gloves

330: Marking of feature points

340: Nail markers

360: active identifier

370: Typing results

380: Hand/finger tracking information

900: system

900B: host system

908: Transceiver

910: Communication link

912: Receiver

906, 916: memory

1110～1180: steps

1210: mobile device

Detailed ways

In the first embodiment, please refer to FIG. 1 , the user interface system 100 can receive data and display information corresponding to the received data. For example in FIG. 1 , the user interface system includes a sensor 110 , a processor 120 , a memory 130 and a display 140 .

The user interface system 100 can receive data and display information corresponding to the received data. For example, the user interface system 100 may be a heads-up display system, such as eyeglasses or any type of near-eye display unit that includes a display.

The sensor 110 detects the movement of the user's finger and the user's hand, and provides the detection result to the processor 120 . The aforementioned user interface system may include one or more sensors coupled to the processor.

The processor 120 can interpret hand/finger movements as data input to the user interface system 100 . Processor 120 may process data to be displayed on a display. The processor 120 may be any type of processor, such as a microprocessor (single-core or multi-core), a digital signal processor (DSP), a graphics processing unit, and the like.

The storage 130 is a built-in data storage of the user interface system 100 . Memory 130 is coupled to processor 120 . The memory 130 can store software accessed and executed by the processor 120 .

The display 140 can be, for example, an optical see-through display, an optical see-around display, a video see-through display, a liquid crystal display (LCD), a plasma display, etc., or Other types of displays. Data may be received between the display 140 and the processor 120 in a wired or wireless manner.

Referring to FIG. 2A , an embodiment of the user interface system 200 is a wearable computing device, such as a head-mounted display (HMD) including glasses. User interface system 200 includes display 202 , on-board computer system 204 , video camera 206 and sensor 208 .

The display 202 can overlay computer-generated graphics on the user's view of the physical world. That is, if the display 202 displays a virtual keyboard and the user's hands are visible to the user, the user can see both the virtual keyboard and his/her hands.

As shown in FIG. 2A, a single display 202 is located at the center of one lens of the glasses. However, the present application is not limited thereto. For example, display 202 may be located at other locations. In addition, the user interface system 200 may include more than one display, for example, providing a second display on another lens of the glasses, or multiple displays on the same lens. Display 202 is connected to and controlled by computer system 204 .

The built-in computer system 204 is, for example, located on the bridge of the glasses or other locations (eg, on the nose of the glasses, etc.), but is not limited thereto. The built-in computer system 204 may include a processor (eg, processor 120 in FIG. 1 ) and memory (eg, memory 130 in FIG. 1 ). On-board computer system 204 may receive and analyze data from camera 206 (as well as data from other sensing devices and/or user interfaces) and control display 202 accordingly. In addition, graphic data, video data, image data, text data, etc. from other data sources can be relayed from the built-in computer system 204 to the display 202 .

The camera 206 is mounted eg on the frame of eyeglasses or elsewhere. The camera 206 can capture images at different resolutions and/or at different frame rates. The camera can be a small-sized camera, such as those used in mobile phones, web cameras, etc., and can be incorporated into the user interface system 200 . It should be understood, however, that the embodiments described herein are not limited to any particular type of video camera. In addition to configuring a single camera 206 as shown in FIG. 2A, multiple cameras can also be configured, and each camera can be configured to capture the same scene, or capture different angles/scenes, and thus can be installed in different areas of the glasses. The camera 206 may be a two-dimensional (2D, two dimensional) camera or a three-dimensional (3D) camera.

The content captured by the camera 206 may be similar to the field of view of the user. Other configurations are also possible. As shown in FIG. 2A, the camera 206 may be mounted on the bridge arm of the glasses. In other embodiments, the camera 206 may be mounted on the nose of the glasses or near the user's forehead/eyes. Camera 206 may be oriented to be in the same direction as the user's eyes, taking images in front of the glasses.

The display 202 is positioned and sized so that the displayed image appears to be the user's view of the "floating" in the real world, thereby providing the user with a sense that computer-generated information can be compared to the real world. The user perception of the world merges. In doing so, the on-board computer system 204 can analyze the images captured by the camera 206 to determine which images should be displayed and how the images should be displayed (eg, where on the display, at which size they are displayed, etc.).

The sensor 208 is mounted on the bridge arm of the glasses, however, the sensor 208 may be mounted in other areas of the user interface system 200 . Additionally, additional sensors may be included in the user interface system 200 .

FIG. 2B illustrates an example operation of the user interface system 200 in 2A. In one embodiment, the user may wear the user interface system 200 as if wearing glasses. Figure 2C schematically illustrates an example of the user interface system in 2A. As shown in FIG. 2C , a virtual input pattern 240 may be displayed on the display 202 of the glasses.

The virtual input pattern 240 may include a QWERTY-like keyboard 241 and a control area 242 . Pattern 240 is sent from on-board computer system 204 to display 202 .

In this example, the QWERTY-shaped keyboard 241 can be divided into a plurality of subgroup keys, and each subgroup key corresponds to a corresponding area of the control area 242 according to a predetermined corresponding relationship.

As shown in FIG. 2C, the QWERTY-shaped keyboard 241 can be divided into 9 subgroup keys. The relationship between the QWERTY-shaped keyboard 241 and the subgroup keys of the 9 areas of the control area 242 can be, for example, a key-area correspondence table, as shown in Table 1 below.

Table 1

In the above, the user's first finger, second finger, third finger, fourth finger and fifth finger corresponding to the key may be, for example, the user's right thumb, index finger, middle finger, ring finger and little finger.

In another example, the user's first finger, second finger, third finger, fourth finger and fifth finger corresponding to the key may be, for example, the user's left little finger, ring finger, middle finger, index finger and thumb respectively.

The subgroup keys of the keyboard correspond to respective areas, according to the relationship between the positions of the subgroup keys of the QWERTY-like keyboard 241 and the positions of the respective areas of the control area. For example, subgroup key "qwert" in the upper left corner on the keyboard corresponds to zone "0" in the upper left corner of the control area. Even, according to the respective positions of these subgroup keys on the virtual keyboard and the respective positions of these areas on the control area, each of these subgroup keys configured on the virtual keyboard corresponds to one of the areas in the control area. Embodiments of what the above tables disclose and how they are entered in the user interface system 200 are described below.

2A-2C illustrate an embodiment of a user interface system including glasses. Additionally, components of the system can be separated from the glasses. For example, the camera 206 and/or sensor 208 can be separate from the glasses or be removable, and can be attached to other body parts of the user, including the neck, chest, wrist or (waist) belt, etc., and can extract images, And sense the user's hand/finger movement.

Referring to FIG. 3A-FIG. 3D, it shows an embodiment of inputting (air typing) at the user interface system, wherein the user may input data while wearing a marker or gloves. When the user wants to type, the user puts his/her right hand within range of the camera and sensor. In this example, before air typing, the user wears a glove 320 with feature point markers 330 and five nail-shaped markers 340 . When the user wears the glove 320, the position of the feature point marker 330 is relative to the feature point of the user's hand. The feature point is, for example but not limited to, any point on the back of the user's hand. For example, the feature point includes the mouth of the user's hand or the finger pit of the user's hand or the center point of the back of the user's hand. Nail markers 340 are associated with the nails of the user's hand 310 . For example, the nail-shaped marker 340 can be a ring, a removable tape that can be adhered to the nail/finger, or nail polish that is rubbed on the nail. In the following description, the position of the feature point mark 330 is relative to the back of the user's hand.

When inputting, the user extends his/her right hand 310 until the built-in computer system 204 determines that the feature point marker 330 of the glove 320 is within the selected area of the control area. For example, as shown in FIG. 3A, the user interface system 200 determines whether the feature point marker 330 of the glove 320 has been located within zone "3" within the control area. If so, the user interface system 200 determines that the user wants to enter a key among a, s, d, f and g.

For the built-in computer system 204 to determine, the sensor 110 or the camera 206 senses/extracts the feature point marker 330 and the nail-shaped marker 340 , and sends the sensing/extracted result to the built-in computer system 204 .

Based on the extracted images, after the built-in computer system 204 extracts the images of the feature point marker 330 and the nail-shaped marker 340 , the built-in computer system 204 obtains the coordinates of the feature point marker 330 and the nail-shaped marker 340 . The built-in computer system 204 compares the coordinates of the feature point marker 330 with the area coordinates of the control area to determine the area where the feature point marker 330 is located.

After the built-in computer system 204 determines the selected area where the feature point marker 320 is located, the built-in computer system 204 controls the display 202 to display an active identifier 360 . The activity identifier 360 shows which keys correspond to the selection area where the feature point marker 330 is located. This active identifier 360 can be highlighted so that the user can easily see which keys are corresponding (or active). Thus, a user can type/input on the user interface system 200 .

For example, if the user wants to input the key "d" corresponding to the area "3", the user moves his/her right hand until the built-in computer system 204 of the user interface system 200 determines that the feature point mark 330 of the user's right hand 310 is located at the control "3" within the region of the region. On-board computer system 204 then controls display 202 to display activity identifier 360 . After displaying the active identifier 360 (surrounding the keys a, s, d, f and g), the user moves/tap/bends the middle finger. The built-in computer system 204 interprets this finger movement as hitting the key "d". By recognizing/translating this finger hitting the key "d", the built-in computer system 204 controls the display 202 to display the typing result 370, which shows "d", and sends a typing event representing the key "d" being hit to the host ( not shown).

Similarly, as shown in FIG. 3B, if the user wants to input "h", the user moves the glove 320 until the built-in computer system 204 of the user interface system 200 determines that the feature point mark 330 of the user's right hand 310 is located in the area of the control area "4". When the user sees the active identifier 360, the user moves/tap/bends the index finger to indicate hitting the key "h". By recognizing the finger tap as hitting the key "h", the embedded computer system 204 controls the display 202 to display the typing result 370 (which displays "dh"), and the embedded computer system 204 sends a typing event indicating The key "h" is struck. The operations in Fig. 3C and Fig. 3D are similar, and the description is omitted for brevity.

In addition, after the built-in computer system 204 obtains the coordinates of the nail-shaped marker 340, the built-in computer system 204 controls the display 202 to display hand/finger tracking information 380, which represents the position of the nail-shaped marker 340 of the glove 320 (i.e. position of the nails of the user's hand 310).

Because the correspondence between the layout of the QWERTY keyboard 241 and the control area 242 is designed based on human instinct, the user can memorize the layout of the QWERTY keyboard 241 and the corresponding relationship with little or no psychological burden. For example, keys q, w, e, r and t (in the upper left corner of the QWERTY-shaped keyboard 241) correspond to zone "0" in the upper left corner of the control area 242; , h, j, k and l correspond to the middle area "4" of the control area 242 .

Referring to FIG. 4 , there is shown another schematic illustration of a keyboard layout in which a user may wear a marker or gloves to enter data according to an embodiment of the present disclosure. In this example, the QWERTY-like keyboard can be divided into 12 subgroup keys. Likewise, the relationship between the 12 subgroup keys of the QWERTY-shaped keyboard and the individual areas of the control area can be summarized in the key-area correspondence table, as shown in Table 2. In other possible embodiments of the present disclosure, the user may enter data with the marker worn on the finger (but the user does not wear gloves). In other possible embodiments of the present disclosure, a user may enter data wearing a glove or set of gloves that includes feature points but no markings. That is to say, in the present disclosure, the glove, the nail type mark and the feature point mark are optional.

Table 2

The user operation of the keyboard layout shown in FIG. 4 is similar to that in FIG. 1 , so for the sake of brevity, details are omitted here.

Referring to FIG. 5 , it shows another embodiment according to the present disclosure, wherein the user can input data with bare hands. As described above, in FIGS. 3A-3D and FIG. 4 , the user wears a glove with feature point markers 330 and five nail-shaped markers 340 to enter data into the user interface system. The difference is that in FIG. 5 , when the user is air typing on the user interface system 200 , the user does not wear gloves. Therefore, the user interface system 200 detects and tracks the feature points of the user's hand to determine whether the feature points of the user's hand are located within the selected area of the control area. Similarly, the feature point can be any point on the back of the user's hand or the mouth of the tiger. Additionally, the user interface system 200 detects and tracks the fingertips of the user's hand, and controls the display to display hand/finger tracking information.

The user operation of the keyboard layout shown in FIG. 5 is similar to that in FIG. 1 , so for the sake of brevity, details are omitted here.

Referring now to FIG. 6, another example of an embodiment of the present disclosure is shown. In Fig. 6, the user does not wear gloves with markings when typing. That is, the user can input data with both hands (empty hands).

In FIG. 6 , the user uses two hands to input data into the user interface system 200 . As shown in FIG. 6 , the QWERTY-shaped keyboard can be divided into 12 subgroup keys, and each subgroup key corresponds to a corresponding area of the control area according to a predetermined corresponding relationship. Likewise, the relationship between the 12 subgroup keys of the QWERTY-shaped keyboard and the 12 areas of the control area can be summarized in a key-area correspondence table such as Table 3. Additionally, the left half of the control area (e.g., areas 0, 1, 4, 5, 8, and 9) corresponds to the keys that would be pressed by the user's left hand; the right half of the control area (i.e., areas 2, 3, 6, 7, 10, and 11) correspond to keys that would be pressed by the user's right hand.

table 3

After the user interface system determines that the feature point of either hand of the user enters an area of the control area, the active identifier is displayed, and the user taps/bends with left/right hand fingers to enter the user interface system 200 .

For example, if the user wants to enter "a" and "&", the user's left hand moves to zone "4" and taps/bends the left pinky finger; then the user moves the right hand into zone "7" and taps/bends the right thumb.

Referring now to FIG. 7 , another embodiment of the present disclosure is shown. In Fig. 7, when typing, the user does not wear gloves with markings, and the control area can be combined with subgroup keys of keys, and the corresponding keys are changed by hand movements.

The QWERTY-like keyboard in FIG. 7 can be divided into 6 subgroup keys. As shown in the layout of Figure 7, subgroups of keys overlap the area of the control area. In other words, in the layout example of FIG. 7, the areas of the control area are combined with subgroups of keys set to the virtual keyboard configuration, the left subgroup of keys (fields) assigned to the user's left hand, and the right subgroup of keys (fields) assigned to the user's left hand. Subgroup keys (fields) are assigned to the user's right hand. After the user interface system 200 detects and determines that the feature point of the user's left/right hand enters any subgroup key (area), the user interface system 200 controls the display to display the active sign surrounding the selected area entered by the feature point symbol. For example, if the user interface system 200 detects and determines that the feature point of the user's left/right hand enters the upper left subgroup key (upper left area), the user interface system 200 controls the display to display the image surrounding the upper left area (which corresponds to the key "qwert"). Activity identifier. Once the active identifier is seen, the user can tap/bend their fingers to type.

Furthermore, in Figure 7 of this example, as soon as the user's hand is detected to be within the subgroup key/area, the user can tap/bend the finger to start typing. For example, if the user interface system detects and determines that a feature point of the user's hand enters within a range corresponding to the "asdfg" subgroup key/region, the user interface system determines that the user is about to type the left middle subgroup key (including the key " asdfg"). Likewise, if the user interface system detects and determines that a feature point of the user's hand enters the subgroup key/area corresponding to the key "nmSB.", the user interface system determines that the user is about to type the right bottom subgroup key (including key "nmSB.").

Referring now to FIG. 8A and FIG. 8B , it shows another example of the embodiment of the present disclosure, wherein the control area can be combined with subgroup keys, and the corresponding keys can be changed by hand movements. As shown in FIGS. 8A and 8B , the user does not wear gloves with markings when inputting. A key layout different from that of FIG. 7 is illustrated in FIGS. 8A and 8B . The operation of FIG. 8A and FIG. 8B is similar to that of FIG. 7, so the details are not repeated here. In FIGS. 7 , 8A, and 8B, the feature points include corresponding points on the back of the user's right/left hand.

As shown in FIGS. 6-8B , the user is not wearing marked gloves. Additionally, the user may wear gloves with markers to assist/track feature points as well as the user's fingers while the user is typing the examples in Figures 6-8B.

Additionally, in other possible embodiments, the keyboard layout may be adapted for one-handed typing, other than the examples shown in Figure 7, Figure 8A, or Figure 8B where the user is typing with both hands. For example, the subgroup keys of the keyboard layout are arranged in an order (eg, a top-to-bottom order) such that the user can input with one hand. This is still within the spirit and scope of the invention. In one embodiment, the respective positions of these areas of the control area can be superimposed on the respective positions of the subgroup keys of the virtual keyboard configuration.

In the above example, the keyboard layout, control area, activity identifier, typing result and hand/finger tracking information are displayed on a display in the user interface system. The image obtained by extracting the user's one hand/both hands is not necessarily displayed on the display of the user interface system, because the user can see the user's one hand/both hands with eyes.

FIG. 8C discloses one embodiment of the present disclosure, another example where a user can use a single finger for input. As shown in FIG. 8C , the system determines that the feature point is "3" located in the area, and the system will display the active identifier 360, which surrounds the key "asdfg". The system detects the position of the user's finger (such as but not limited to the index finger), the system determines that the user wants to enter "d", then the system will display another active identifier 390, which surrounds the key "d". The user may then tap his finger, and the system will interpret the fingertip's press of the key "d".

To disclose another embodiment where the user can input with a single finger. As shown in this Figure 8D, when the system then determines that the feature point is "3" located in the region, the system displays the activity identifier 360 surrounding the key "asdfg". After the system detects the location of the user's finger (such as but not limited to the index finger), the system also displays the key "asdfg" 395, which is located near the location of the user's finger. Afterwards, the system determines that the user wants to enter "d", then the system displays another active identifier 397 surrounding the key "d" of the key "asdfg" 395. The user may then tap his finger, and the system will interpret the fingertip as pressing the key "d". Similarly, FIG. 8E shows that the user may move the user's finger to strike the key "a".

An example of a user typing with one finger is shown. Of course, the user can type with any finger of either hand. in other words, can be with Combinations of the embodiments of the figures are still within the scope of the present disclosure.

Referring now to FIG. 9 , there is shown a system 900 for receiving user input according to another embodiment of the present disclosure. The system 900 includes a user interface system 900A and a host system 900B. User interface system 900A includes sensor 902 , processor 904 , memory 906 , and transceiver 908 . Host system 900B includes receiver 912 , processor 914 , memory 916 , and display 918 . User interface system 900A and host system 900B are coupled via a communication link 910 . Communication link 910 may be a wireless or wired connection. For example, communication link 910 may be a wired connection, such as a serial bus such as USB, or a parallel bus. The communication link may be a wireless connection, such as Bluetooth, IEEE802.11 or other wireless communication links.

Components 902, 904, 906 of user interface system 900A are similar to user interface systems 100/200, except that user interface system 900A additionally includes a display. The transceiver 908 of the user interface system 900A is used to send and receive data from the host system 900B. The receiver 912 of the host system 900B is used to send and receive data from the user interface system 900A.

The functions and operations of processor 120 of user interface system 100 may be performed by processor 904 and/or processor 914 . For example, the processor 904 determines whether the feature point of the glove/user's hand is located within a selected subgroup of keys/areas, and interprets the user's finger tap as a key press. Through the transceiver 908 , communication link 910 and receiver 912 , the decision results of the processor 904 and the captured images may be sent to the processor 914 . The processor 914 controls the display 918 to display keyboard layouts, control areas, images of the user's hands, activity identifiers, typing results, hand/finger tracking information, and the like.

An example of the system 900 of FIG. 9 is shown in FIG. 10 . As shown in FIG. 10, the user interface system 900A can be a wearable computing device, and the host system 900B can be any type of computing device, such as a PC, laptop computer, mobile phone, etc., that transmits data via wired or wireless The mode is transmitted to the user interface system 900A.

After the user wears the user interface system 900A, the user can extend the hand/glove to the sensing range of the sensor 902 and/or the extraction range of the image capture device; and the user can see the image of the hand displayed on the display 918 of the host system 900B.

After the processor 904 of the user interface system 900A determines that the feature point of the hand/glove has been located within the subgroup key/area, the active identifier is displayed on the display 918 of the host system 900B, and the user can tap/flex the finger to type in user interface system 900A. The typing result is displayed on the display 918. That is to say, during typing, the user watches the display 918 of the host system 900B.

The keyboard layout and control areas shown can be adapted for use in FIGS. 9 and 10 . Additionally, keyboard layouts, control areas, active identifiers, images of the user's hands (both hands), typing results and hand/finger tracking information can be displayed on the host system's display. On the other hand, in the examples of FIGS. 9 and 10 , the user interface system may be used to capture images of the user's hand (both hands) and to sense movement of the user's hand (both hands)/finger(s). The determination of whether the user's hand (both hands) has been located within the selection area, and the determination of the user's finger tap can be performed by the user interface system 900A or the host system 900B.

Referring to FIG. 11 , it is a method according to another embodiment of the present disclosure. The method of FIG. 11 can be used in the system shown in FIG. 1 or FIG. 9 . Figure 11 may include one or more operations, functions or actions. Although the steps are shown in a sequential order, the steps may be performed in parallel, and/or in a different order. Furthermore, steps may be combined into fewer steps, or split into multiple steps, and/or deleted as desired.

Furthermore, for the methods disclosed herein, the flowchart of FIG. 11 shows one possible implementation of the functionality and operation of this embodiment. In this regard, various steps may be implemented as a module, circuit, program segment or part of program code comprising one or more instructions executable by a processor to implement the disclosed functions or steps. The program code can be stored on any type of computer readable medium, for example, a storage device such as a hard disk or a flash memory. The computer readable medium may include non-transitory computer readable medium, such as random access memory (RAM). The computer readable medium may also include non-transitory media such as read only memory (ROM), optical or magnetic disks, or built-in memory of firmware. The computer readable medium can also be any other volatile or nonvolatile storage system.

In step 1110, the keyboard layout and control areas are displayed. For example, the keyboard layout and control areas may be displayed on the user interface system and/or the display of the host system.

In step 1120, images of the user's hands (both hands) are extracted. For example, images of the user's hands (both hands) may be captured by the camera of the user interface system. The order of steps 1110 and 1120 are interchangeable.

In step 1130, finger images, fingertip images and/or feature point images are extracted from the extracted images. For example, finger images, fingertip images, and/or feature point images may be extracted by the user interface system and/or processor(s) of the host system. The step of extracting a plurality of object images may include extracting a plurality of fingers, a plurality of fingertips, and/or at least one feature point from, for example, a user's hand, an image obtained from a plurality of images of the object in at least the extracted image. .

In step 1140, the selection area where the feature points are located is determined. For example, the user interface system and/or the processor(s) of the host system may determine a selected area of the control area where the feature point is located.

At step 1150, the number of keys corresponding to the user's fingers is determined, eg, by the user interface system and/or the processor(s) of the host system. Key mapping refers to mapping the key to the selection area where the feature point is located, and/or, mapping the key to the user's finger. As described in Figure 6 above, for example, if the feature point is located in the area "4" and the key "asdfg" corresponds to the user's finger, the key correspondence means that the key "asdfg" is mapped to the area "4".

At step 1160, the user finger taps are interpreted as key taps, eg, by the user interface system and/or the processor(s) of the host system.

At step 1170, after interpreting the finger strokes, the typing results and hand/finger tracking information are transmitted and displayed on, for example, a display of the user interface system and/or the host system.

In step 1180, it is determined whether the user has terminated or interrupted the input of data. For example, the user can terminate or interrupt data input by voice control or by pressing a physical or virtual key. If step 1180 is no, the process returns to step 1110 . If yes in step 1180, the process ends.

Fig. 12 illustrates a system for receiving user input according to yet another embodiment of the present disclosure. As shown in FIG. 12, during data entry, the user is holding a mobile device 1210 (such as, but not limited to, a mobile phone or tablet computer). The mobile device 1210 includes similar or identical sensors, processors, memory, and displays as in FIG. 1 . As the user enters, the mobile device 1210 picks up and senses one of the user's hands. The extracted images of hands, typing results, virtual keyboard configurations and control areas are displayed on the mobile device 1210. The operation of the user inputting data into the mobile device 1210 is the same or similar to that discussed above, so it is omitted here.

The present disclosure also discloses a program storage medium, the computer program stored in it can cause the electronic device to perform the following steps: display a virtual keyboard configuration and a control area, the virtual keyboard configuration includes a plurality of subgroup keys, each subgroup The set of keys corresponds to one of a plurality of regions of the control region; an image of an object is taken out from at least one extracted image to identify the position of a feature point of the object; and the feature point is determined to be located among the regions of the control region a selected area; determining keys corresponding to the selected area; and interpreting movement of the object as an input data to a user interface system.

In addition, in one embodiment, the computer system further includes at least one processor, configured to execute related control programs.

In another embodiment, the computer system may be a circuit design that may be implemented on a chip. Specifically, the embodiments may be implemented as circuit designs, including integrated circuits and layouts, by a hardware description language (such as Verilog or VHDL, etc.). Circuits can be designed using a variety of hardware description languages. For example, integrated circuit manufacturers may implement with application-specific integrated circuits (ASICs) or customer-oriented integrated circuits.

As mentioned above, because the areas of the control area are very close to each other, the embodiment of the present disclosure is characterized in that the hand movement distance can be minimized during condensate typing. On the contrary, when typing on a traditional keyboard, the hand movement distance is usually long, so that the user may need to spend more time typing.

In summary, although the present application has been disclosed as above with embodiments, it is not intended to limit the present application. Those skilled in the art to which this application belongs may make various changes and modifications without departing from the spirit and scope of this application. Therefore, the scope of protection of the present application should be determined by the appended claims.

Claims (18)

1. a kind of reception user's input method, comprises the following steps：

Show dummy keyboard configuration and control area, dummy keyboard configuration includes multiple subgroup keys, each subgroup key difference Correspond to one of multiple regions of the control area；

Multiple positions of object are extracted from least one extraction image, identify the position of the characteristic point of the object；

It determines chosen area, is the selection area among all areas in the control area, where the position of this feature point Domain；

Determine multiple keys corresponding to the chosen area；And

The movement of the object is interpreted as the input data to user interface system.

2. input method as described in claim 1, wherein each self-alignment configured according to these subgroup keys in the dummy keyboard The respective position in the control area with these regions is put, these each subgroup keys which is configured correspond to the control One of these regions in region processed.

3. input method as described in claim 1, wherein the step of multiple positions of the extraction object include：

From at least one extraction at least image for multiple fingers of image zooming-out user, an at least image for multiple finger tips and/ Or at least image of this feature point of proficiency.

4. input method as described in claim 1, the wherein object are with or without the empty-handed of mark,

This feature point includes the extremely a little less of the back of the hand of at least proficiency of user, and

The user this at least the back of the hand of proficiency should be to a little less at least tiger's jaw of proficiency or the back of the hand including the user Center.

5. input method as described in claim 1, the wherein object are one or more gloves, marked including characteristic point and more A nail phenotypic marker object；And

The step of multiple positions of the extraction object, includes：

Go out the multiple of at least one position and these nail phenotypic markers of this feature point mark from at least one extraction image zooming-out Position.

6. input method as described in claim 1, further comprises the steps：

The active identifier of display, around these keys for corresponding to the chosen area.

7. input method as described in claim 1, further comprises the steps：

Display at least an input results, tracked information and the object this at least one extraction image it is at least one.

8. this of these regions of input method as described in claim 1, the wherein control area and dummy keyboard configuration A little group's bond closes setting or the respective position in these regions of the control area and these subgroups of dummy keyboard configuration The respective position of group key coincides setting.

9. a kind of reception user input systems, the system include：

Display, for showing dummy keyboard configuration and control area, dummy keyboard configuration includes multiple subgroup keys, each son Group's key corresponds to one of multiple regions of the control area respectively；

One or more sensors, the movement of sensing article；

Computing system, is coupled to one or more sensors and the display, which carries from least one extraction image Multiple positions of the object are taken, to identify the position of the characteristic point of the object, chosen area are determined, in these of the control area Among region, the position where this feature point is the chosen area, determines to correspond to multiple keys of the chosen area and incite somebody to action The movement of the object is interpreted as the input data to the system.

10. input system as claimed in claim 9, the wherein computing system are matched somebody with somebody according to these subgroup keys in the dummy keyboard The respective position put and these regions control the display to show dummy keyboard configuration in the respective position of the control area These each subgroup keys correspond to one of this few region of the control area.

11. input system as claimed in claim 9, when wherein the computing system extracts the position of the object, from this at least one Image is extracted, extracts at least the one of an at least image for multiple fingers of user, an at least image for multiple finger tips and the user The image of this feature point of hand it is at least one.

12. input system as claimed in claim 9, the wherein object are with or without the empty-handed of mark,

This feature point includes the extremely a little less of the back of the hand of at least proficiency of user；And

The back of the hand of at least proficiency of the user should be to a little less this including the user at least in the tiger's jaw of proficiency or the back of the hand The heart.

13. input system as claimed in claim 9, wherein

The object is one or more gloves, including characteristic point mark and multiple nail phenotypic marker objects；And

The computing system goes out at least one position and these nail types of this feature point mark from at least one extraction image zooming-out Multiple positions of mark.

14. input system as claimed in claim 9, wherein, which controls the display, with display activity mark Symbol, around these keys for corresponding to the chosen area.

15. input system as claimed in claim 9, wherein, which controls the display to show input results, chase after This of track information and the object at least one extracts at least one of image.

16. input system as claimed in claim 9, wherein, which controls the display to show, the control area The configuration of these regions and the dummy keyboard these subgroups bond close set or these regions of the control area it is respective The respective position of these subgroup keys of position and the dummy keyboard configuration coincides setting.

17. a kind of reception user inputs program recorded medium, it is following that the computer program stored includes electronic equipment execution Step：

Show dummy keyboard configuration and control area, dummy keyboard configuration includes multiple subgroup keys, each subgroup key difference Correspond to one of multiple regions of the control area；

Multiple positions of object are extracted from least one extraction image, identify the position of the characteristic point of the object；

It determines chosen area, is the chosen area among the multiple regions of the control area, where the position of this feature point；

Determine to correspond to multiple keys of the chosen area；And

Judge the movement of the object and corresponded to these keys of the chosen area using as the input to user interface system Data.

18. program recorded medium as claimed in claim 17, the computer program stored is used to hold the electronic device Row includes such as claim 2, the method for 3,4,5,6,7 or 8.