CN110109599A - The interactive approach of user and stylus, categorizing system and stylus product - Google Patents

Abstract

The present disclosure proposes a novel method to operate a stylus product, using the signal of an inertial measurement unit (IMU) to calculate the inclination angle of the stylus product, and collecting the acceleration signal measured when the finger taps the stylus to train a deep neural network as a tap classifier. Combining the inclination angle with the tap classifier allows the user to interact with peripheral devices (such as a touch screen) by rotating and tapping the stylus product. The present disclosure also provides a tap classification system and a stylus product.

Description

User interaction method with stylus, classification system, and stylus product

technical field

The present disclosure relates to a human-computer interaction technology, and in particular, to a method for interacting between a user and a stylus, a classification system for tapping events performed by a user on the stylus, and a stylus product.

Background technique

As an indicator tool, a stylus is usually used in conjunction with a touch screen to realize the function of selecting objects displayed on the screen. Due to increased screen resolution and improvements in styluses, styluses are also commonly used as writing tools, or as brushes. On the other hand, various grades of drawing or writing tablets also require the use of a stylus. However, the function of the stylus is still weak at present, so developing a new generation of stylus to increase its function and product competitiveness has become one of the important issues.

SUMMARY OF THE INVENTION

The purpose of the present invention is to provide a method for interacting between a user and a stylus, a classification system for tapping events performed by a user on the stylus, and a stylus product, so as to provide a novel interaction method.

In order to achieve the above object, one aspect of the present invention provides a method for interacting between a user and a stylus. The method includes: using a sensor to sense various taps generated by a user tapping a stylus. Click events to measure several acceleration signals; sample each acceleration signal, and obtain several eigenvalues for each acceleration signal; combine these eigenvalues of an acceleration signal with the corresponding tapping events according to the acceleration signal The classification mark of the type of record is used as a sample to generate a sample set containing several samples; these eigenvalues in a sample are used as input, and the freely selected weight parameter group is used as an adjustment parameter, which is input into a deep neural network. generate a predicted classification mark; according to the error between the predicted classification mark and the real classification mark in the sample, adopt the algorithm of backward propagation to adjust the weight parameter group; and read out the samples of the sample set in batches to train The deep neural network fine-tunes the weight parameter group to determine an optimized weight parameter group.

Another aspect of the present invention provides a system for classifying tapping events performed by a user on a stylus pen, the system includes: a stylus pen with a sensor disposed therein for sensing the touch by the user. Various tapping events generated by tapping the stylus to measure several acceleration signals; and a computer device coupled to the stylus, the computer device comprising: a processor for receiving the sensor the transmitted acceleration signals; and a memory connected to the processor, the memory containing a number of program instructions executable by the processor, the processor executing the program instructions to perform a method, the method comprising: Each acceleration signal is sampled, and several eigenvalues are obtained for each acceleration signal; these eigenvalues of an acceleration signal and the classification marks recorded according to the type of the tapping event corresponding to the acceleration signal are used as a sample to generate a sample containing A sample set of several samples; these eigenvalues in a sample are used as input, the freely selected weight parameter group is used as adjustment parameters, and input into a deep neural network to obtain a predicted classification label; according to the predicted classification The error between the label and the real classification label in the sample is adjusted using a backward propagation algorithm to adjust the weight parameter group; and the samples of the sample set are read out in batches, the deep neural network is trained, and the weight parameter group is fine-tuned , to determine the optimal weight parameter group.

Another aspect of the present invention provides a stylus product, the stylus product includes: a sensor for sensing an acceleration signal generated by a tapping operation on the stylus product, and used for The inclination angle of the stylus product is calculated by a fusion algorithm; a controller is coupled to the sensor, a deep neural network corresponding to the deep neural network of the aforementioned method is built in the controller, and the controller is used for The corresponding deep neural network and the optimized weight parameter set obtained according to the aforementioned method are used as a knock classifier, and the acceleration signal from the sensor is input into the knock classifier to obtain a a predicted tap type; and a wireless transmission module coupled to the controller for transmitting a wireless signal carrying the predicted interaction type and the calculated tilt angle.

The present disclosure deploys a sensor, its fusion algorithm, and a tap classifier in a stylus product to provide a novel way of operation. In some embodiments, the user can perform many convenient interactions by turning and tapping the stylus product without directly or indirectly touching the touch screen.

In order to make the above-mentioned content of the present disclosure more obvious and easy to understand, the preferred embodiments are exemplified below, and are described in detail as follows in conjunction with the accompanying drawings.

Description of drawings

FIG. 1 shows a schematic diagram of a user holding a stylus.

FIG. 2 shows a schematic diagram of a classification system for tapping events implemented according to an embodiment of the present disclosure.

FIG. 3 shows a flowchart of a method for interacting a user with a stylus according to an embodiment of the present disclosure.

FIG. 4 shows a schematic diagram of a deep neural network in an embodiment of the present disclosure.

FIG. 5 shows a schematic diagram of a stylus product implemented according to an embodiment of the present disclosure.

FIG. 6 shows a flowchart of a method for interacting a user with a stylus according to an embodiment of the present disclosure.

FIG. 7A shows a schematic diagram of an adjustment bar displayed on a screen in an embodiment of the present disclosure.

FIG. 7B shows a schematic diagram of a color pie chart displayed on a screen in an embodiment of the present disclosure.

FIG. 7C shows a schematic diagram of a coordinate system of the stylus product in the embodiment of the present disclosure.

Detailed ways

In order to make the objectives, technical solutions and effects of the present disclosure clearer and clearer, the present disclosure will be described in further detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present disclosure, and the word "embodiment" used in the specification of the present disclosure is intended to be used as an example, illustration or illustration, and is not used to limit the present disclosure. In addition, as used in this disclosure and the appended claims, the article "a" can generally be construed to mean "one or more" unless specified otherwise or clear from the context in the singular form.

FIG. 1 shows a schematic diagram of a user holding a stylus. The user's hand can interact with the stylus 10 , such as changing the inclination angle of the stylus 10 and tapping the stylus 10 with fingers. The purpose of the present disclosure is to provide a novel interactive manner, allowing the user to rotate and tap the stylus 10 to interact with peripheral devices (eg, a touch screen).

The present disclosure adopts the sensor fusion technology, combined with the tapping classifier obtained through deep learning, so that the user can perform various tasks by changing the inclination angle of the stylus 10 and tapping the stylus 10. an operation. The sensor fusion technology can use a six-axis or nine-axis inertial measurement unit (IMU) to calculate the inclination angle of the stylus 10 through a fusion algorithm (eg, Madgwick's algorithm or Kalman filter). In addition, a tapping classifier is obtained by classifying and learning the tapping events generated by the user tapping the stylus 10 . Using the stroke classifier, the user's strokes of the stylus 10 can be classified.

The type of changing the tilt angle may include at least one of a roll angle for rotating the stylus 10 along the X axis, a pitch angle for rotating along the Y axis, and a yaw angle for rotating the Z axis First, the right-hand rule can be used to determine the positive and negative directions of the rotation angle (see Figure 7C).

The type of tap event may, for example, depend on the type or number of times the user's finger taps the stylus 10, such as one tap, two taps, three taps, and the like.

Combining the inclination angle of the stylus and the tap classification technology, it can be applied to many application scenarios, such as rotating the stylus to change the color or font size of the stroke, and then tapping the stylus with a finger to confirm or cancel, which can be determined according to the Different application scenarios have different configurations. Those skilled in the art will appreciate that the inventive concepts of the present disclosure can also be applied to other applications. Of course, the relationship between the types of interactive events and the operations performed can also be defined by the user.

FIG. 2 shows a schematic diagram of a classification system for tapping events implemented according to an embodiment of the present disclosure. The system includes a stylus 10 and a computer device 40 coupled to the stylus 10 . The stylus 10 can be a capacitive stylus, which can change the capacitance on a touch panel (not shown) to generate touch operations. The computer device 40 can be a calculator with certain computing capabilities, such as a personal computer, a notebook computer, and the like. In the present disclosure, when classifying the tapping events between the user and the stylus 10, the tapping events need to be collected first. Here, various actions are manually performed on the stylus 10, and the signals corresponding to the tapping events It is transmitted to the computer device 40, and the computer device 40 performs learning using a deep neural network.

As shown in FIG. 2 , the stylus 10 includes at least one sensor 20 . The sensor 20 may be a (nine-axis) IMU, such as a combination of a three-axis accelerometer, a three-axis gyroscope, and a three-axis magnetometer. Depending on the application, the sensor 20 may also be a combination of a (triaxial) accelerometer and a (triaxial) gyroscope. The sensor 20 may be disposed at any position within the stylus 10 .

The computer device 40 receives the acceleration signal generated by the sensor 20 through an interface, and feeds it into the deep neural network for classification learning. After the tapping events are generated manually, the type of each tapping event can also be input into the computer device 40 to perform supervised learning. As shown in FIG. 2 , the computer device 40 includes a processor 41 and a memory 42 , the processor 41 receives the acceleration signal from the sensor 20 , the memory 42 is connected to the processor 41 , and the memory 42 includes the functions executable by the processor 41 . Several program instructions, the processor 41 executes the program instructions to perform the relevant operations of the deep neural network. The computer device 40 can also use GPU or TPU to perform the related operations of the deep neural network, so as to improve the operation speed.

FIG. 3 shows a flowchart of a method for interacting a user with a stylus according to an embodiment of the present disclosure. Please refer to FIG. 3 in conjunction with FIG. 2 , the method includes the following steps S31 to S36 , and these steps correspond to the training process of the tapping classifier.

Step S31 : Use the sensor 20 to sense various tapping events generated by the user tapping the stylus 10 to measure several acceleration signals. In this step, the stylus 10 is artificially interacted to generate various tap events, and the sensor 20 disposed in the stylus 10 senses the tap events to generate acceleration signals. In the present disclosure, the number of the sensor 20 is not limited to one, but may also be several, and the sensor 20 may also be disposed at any position in the stylus 10 .

In one embodiment, the sensor 20 may be implemented with a nine-axis IMU. The acceleration detected by the sensor 20 is a function of time and has three directional components, and the three directional components can be projected into the frequency space respectively by using Fourier transform. In particular, the method may further comprise the step of transforming each acceleration signal from a time distribution to a frequency space.

After converting to frequency space, low-frequency DC components and high-frequency noise can be further filtered out to avoid the influence of gravitational acceleration and noise on the classification result. Specifically, the method may further comprise the step of filtering each acceleration signal, filtering out high frequency and low frequency components.

In another embodiment, the sensor 20 may be implemented using a three-axis accelerometer and a three-axis gyroscope. The three-axis accelerometer can measure three acceleration components of the stylus 10 moving in the three-dimensional space, thereby obtaining an acceleration signal.

Step S32 : sampling each acceleration signal, and obtaining several feature values for each acceleration signal. In this step, each acceleration signal generated by the sensor 20 is sampled. For example, in the frequency space, sampling is performed at certain frequency intervals to obtain several data points. These data points are eigenvalues, which can be found in After normalization, it is used as training data for deep neural network.

Step S33 : Use these characteristic values of an acceleration signal and a classification label recorded according to the type of the tapping event corresponding to the acceleration signal as a sample, and generate a sample set including several samples. In this step, the acceleration signal measured by the sensor 20 and the type of the tapping event corresponding to the acceleration signal are regarded as a piece of data, that is, a sample, and a plurality of samples constitute a sample set. Specifically, a sample contains a characteristic value of an acceleration signal and a classification label corresponding to the acceleration signal.

The sample set can be divided into a training sample set and a test sample set, the training sample set can be used to train the deep neural network, and the test sample set is used to test the classification accuracy of the neural network model obtained by training.

Step S34 : using these feature values in a sample as input, and the freely selected weighting parameters as adjustment parameters, input into a deep neural network, and obtain a predicted classification mark. The feature value in a sample obtained in step S33 is input from the input layer, and the predicted classification label is output through the deep neural network.

Figure 4 shows an example of a deep neural network. A deep neural network can generally be divided into an input layer, an output layer, and a learning layer between the input layer and the output layer. Each sample of the sample set is input from the input layer, and the predicted classification label output from the output layer. Generally speaking, a deep neural network contains many learning layers, and the number of layers is quite large (for example, 50-100 layers), so deep learning can be realized. The deep neural network shown in FIG. 4 is for illustration only, and the deep neural network of the present disclosure is not limited thereto.

A deep neural network can include multiple convolutional layers, batch normalization layers, pooling layers, fully connected layers, and rectified linear units. , ReLu) and a Softmax output layer, etc. In the present disclosure, an appropriate number of layers can be used for learning to achieve a balance between prediction accuracy and computational efficiency, but it should be noted that too many layers may also lead to a decrease in accuracy. A deep neural network can include multiple cascaded sub-networks, each sub-network is connected to each subsequent sub-network, such as DenseNet (Dense Convolutional Network), to improve prediction by feature re-usage accuracy.

Step S35: According to the error between the predicted classification label and the actual classification label in the sample, a backpropagation algorithm is used to adjust the weight parameter group. The optimization goal of the deep neural network is to minimize the classification error (loss), and the optimization method adopts the back propagation algorithm, that is to say, the prediction result obtained by the output layer is compared with the real value to obtain an error value, and then this The error value is passed back layer by layer to correct the parameters of each layer.

Step S36: Read out the samples of the sample set in mini-batches, train the deep neural network, and fine-tune the weight parameter group to determine an optimized weight parameter group. Each time a batch of subsets is used for training, the weight parameter group is fine-tuned once, and this is done iteratively until the classification error tends to converge. Finally, the parameter group with the highest prediction accuracy for the test set is selected as the optimized model parameter group.

FIG. 5 shows a schematic diagram of a stylus product implemented according to an embodiment of the present disclosure. As shown in FIG. 5 , the stylus product 10' includes one or more sensors 20', a controller 30 and a wireless transmission module 50. The sensor 20' is disposed anywhere within the stylus product 10'. In one example, the sensor 20' can be an inertial measurement unit (IMU), which has built-in three-direction accelerometers and three-axis gyroscopes, and can sense the acceleration and angular velocity of the stylus product 10'. In another example, the sensor 20' may be a nine-axis IMU, including a three-axis accelerometer, a three-axis gyroscope, and a three-axis magnetometer.

In one aspect, the sensor 20' is used for sensing a tap operation on the stylus product 10' to generate an acceleration signal. On the other hand, the sensor 20' is used to calculate the inclination angle of the stylus product through a fusion algorithm such as Madgwick's algorithm or Kalmanfilter. The controller 30 is coupled to the sensor 20', and receives the acceleration signal and the calculated inclination angle generated by the sensor 20'.

The controller 30 is used for classifying and predicting the tapping events of the user tapping the stylus product 10', so as to obtain a predicted tapping type. For example, the controller 30 deploys the same or corresponding deep neural network as the deep neural network used in the above steps S34 to S36, and stores the optimized weight parameter set obtained in the above step S36. The corresponding deep neural network and the optimized weight parameter set can be deployed in the stylus product 10' as a tap classifier. The controller 30 inputs the acceleration signal from the sensor 20' into the knock classifier, and then the classification mark of the corresponding knock event can be obtained, that is, the predicted knock type can be obtained. In this way, the stylus product 10' realizes the classification prediction of tapping events.

Specifically, the corresponding program code of the deep neural network and the optimized weight parameter group can be written into the firmware of the controller 30, and the controller 30 executes the algorithm of the deep neural network to predict the type of the tapping event.

The wireless transmission module 50, connected or coupled with the controller 30, is used to receive the type of the tapping event predicted by the controller 30 and the inclination angle of the stylus product 10' from the sensor 20', and transmit the carrying A wireless signal carrying the inclination angle and the predicted tap type. For example, the wireless transmission module 50 can be a Bluetooth module, which can communicate with a tablet computer (or a notebook computer) and transmit this information to the tablet computer through a Bluetooth signal.

Please refer to FIG. 6 in conjunction with FIG. 5 , the method further includes the following steps S38 to S41, and these steps correspond to the process of deploying the tapping classifier into the stylus product 10' to realize various applications.

Step S38: Deploy the deep neural network and the optimized weight parameter set into a stylus product 10' as a tap classifier. In this step, the stylus product 10' has the tapping classifier, which includes the same or corresponding deep neural network as the deep neural network used in the above steps S34 to S36 and the optimized neural network obtained in the above step S36. Weight parameter group.

Step S39: Receive an acceleration signal generated by a tapping operation on the stylus product, and input the acceleration signal generated by the tapping operation into the tapping classifier to obtain a predicted tapping type . In this step, when the user interacts with the stylus product 10' to generate a tap event, the sensor 20' in the stylus product 10' measures an acceleration signal, and inputs the acceleration signal into the tap classifier, and use the controller 30 to predict the type of tap event.

Step S40: Calculate the inclination angle of the stylus product 10' through a fusion algorithm using the IMU disposed in the stylus product 10'. In this step, the IMU can be a six-axis or nine-axis IMU, and the inclination angle of the stylus product 10' can be calculated by Madgwick's salgorithm or Kalman filter algorithm.

Step S41: Execute a predetermined operation according to the predicted stroke type and the calculated inclination angle. In this step, the wireless transmission module 50 in the stylus product 10 ′ transmits the type of the tapping event and the inclination angle predicted by the controller 30 to, for example, a tablet computer (or a notebook computer) through a wireless signal, Software installed on the tablet (or laptop) performs actions that correspond to the predicted results.

In an example application scenario, when the user taps the stylus product once, it can be used to perform a confirmation operation; when a second tap is performed, it can be used to perform a cancel operation.

In another exemplary application scenario, after the user taps the stylus product, the screen 70 (of the tablet computer) displays an adjustment bar to indicate the thickness of the handwriting of the stylus product, as shown in FIG. 7A When the user further rotates the stylus product, the thickness of the handwriting on the stylus product is changed, and the adjustment process and result can be displayed on the adjustment bar. For example, when the stylus product is rotated clockwise, the handwriting becomes thicker; when the stylus product is rotated counterclockwise, the handwriting becomes thinner. Finally, the user taps the stylus again once or twice to confirm or cancel the setting changes.

In another exemplary application scenario, after the user taps the stylus product, the screen 70 (of the tablet computer) displays a color pie chart to indicate the color of the handwriting of the stylus product, as shown in Section 7B As shown in the figure; when the user further rotates the stylus product, the pointer can be moved to select the color of the handwriting, and the adjustment process and result can be displayed on the color pie chart.

In another exemplary application scenario, after the user taps the stylus product, the interactive operation of the stylus product can be used to control the scroll of a window (eg, a window of Microsoft Word) to scroll up and down the displayed content. For example, when the user rotates in the positive direction along the X-axis of the stylus product (see Figure 7C), the content will scroll up, otherwise, the content will scroll down.

In another example application scenario, the user taps the stylus to enter the control mode. At this time, the touch-sensitive display device displays the corresponding menu, and the user rotates the stylus along the X-axis shown in Fig. 7C. That is, a roll action is performed, and the color of the handwriting can be selected, the thickness of the pen tip can be adjusted, the text selection can be performed, or the window scroll can be scrolled by the change of the roll angle. The user then taps the stylus again to confirm the change, or can double-tap to cancel the change. The above adjustment action can also be implemented by changing the pitch angle, and changing the pitch angle can also be performed corresponding to other types of operations.

The present disclosure adopts the sensor fusion technology combined with the tapping classifier to perform various interactive operations on the stylus product, so that the user can perform convenient interactive operations without directly or indirectly touching the control screen in many situations.

The present disclosure has been disclosed above with preferred embodiments, but it is not intended to limit the present disclosure. Those skilled in the art to which the present disclosure pertains can make various modifications and changes without departing from the spirit and scope of the present disclosure. Therefore, the scope of protection of the present disclosure should be determined by the scope of the appended patent application.

Claims (11)

1. An interaction method between a user and a stylus, wherein the method comprises: Utilize a sensor to sense various tapping events generated by the user tapping a stylus, so as to measure several acceleration signals; Sample each acceleration signal, and obtain several eigenvalues for each acceleration signal; Taking these eigenvalues of an acceleration signal and the classification marks recorded according to the type of the tapping event corresponding to the acceleration signal as a sample, a sample set containing several samples is generated; These feature values in a sample are used as input, and the freely selected weight parameter group is used as adjustment parameters, which are input into a deep neural network to obtain a predicted classification mark; Adjust the weight parameter group according to the error between the predicted classification label and the actual classification label in the sample, using a backward propagation algorithm; and The samples of the sample set are read out in batches, the deep neural network is trained, and the weight parameter group is fine-tuned to determine the optimized weight parameter group. 2. The method according to claim 1, wherein the method further comprises: deploying the deep neural network and the optimized set of weight parameters as a tap classifier into a stylus product; and An acceleration signal generated by a tapping operation performed on the stylus product is received, and the acceleration signal generated by the tapping operation is input into the tapping classifier to obtain a predicted tapping type. 3. The method according to claim 2, wherein the method further comprises: Using an inertial measurement unit disposed in the stylus product to calculate the inclination angle of the stylus product through a fusion algorithm; Wherein, after the step of deriving the predicted interaction type, the method further includes: A predetermined operation is performed according to the predicted stroke type and the calculated inclination angle. 4. The method of claim 3, wherein the predetermined operation is selected from the group consisting of changing the color of the brush and changing the thickness of the brush. 5. The method of claim 3, wherein the tilt angle is formed by at least one of a roll angle around the X-axis, a pitch angle around the Y-axis, and a yaw angle around the Z-axis . 6. The method of claim 1, wherein the deep neural network comprises a plurality of convolutional layers. 7 . The method of claim 1 , wherein the type of the tap event includes the type or number of taps performed on the stylus. 8 . 8. A system for classifying tap events performed by a user on a stylus, wherein the system comprises: a stylus with a sensor disposed therein for sensing various tapping events generated by the user tapping the stylus, so as to measure a plurality of acceleration signals; and A computer device coupled to the stylus, the computer device comprising: a processor for receiving the acceleration signals from the sensor; and a memory connected to the processor, the memory containing a number of program instructions executable by the processor, the processor executing the program instructions to perform a method, the method comprising: Sample each acceleration signal, and obtain several eigenvalues for each acceleration signal; Taking these eigenvalues of an acceleration signal and the classification marks recorded according to the type of the tapping event corresponding to the acceleration signal as a sample, a sample set containing several samples is generated; Taking these eigenvalues in a sample as input, the freely selected weight parameter group as adjustment parameters, and inputting them into a deep neural network to obtain a predicted classification mark; According to the error between the predicted classification label and the actual classification label in the sample, using a backward propagation algorithm, adjust the weight parameter group; and The samples of the sample set are read out in batches, the deep neural network is trained, and the weight parameter group is fine-tuned to determine the optimized weight parameter group. 9. The system of claim 8, wherein the deep neural network comprises a plurality of convolutional layers. 10 . The system of claim 8 , wherein the type of the tap event includes the type or number of taps performed on the stylus. 11 . 11. A stylus product, wherein the stylus product comprises: a sensor for sensing an acceleration signal generated by a tapping operation on the stylus product, and for calculating the inclination angle of the stylus product through a fusion algorithm; a controller, coupled to the sensor, a deep neural network corresponding to the deep neural network in claim 1 is built in the controller, and the controller is used for the corresponding deep neural network and according to claim 1 The obtained optimized weight parameter set is used as a tap classifier, and is used to input the acceleration signal from the sensor into the tap classifier to obtain a predicted tap type; and A wireless transmission module, coupled to the controller, transmits a wireless signal carrying the predicted interaction type and the calculated tilt angle.