WO2017131251A1

WO2017131251A1 - Display device and touch input processing method therefor

Info

Publication number: WO2017131251A1
Application number: PCT/KR2016/000894
Authority: WO
Inventors: 김태호; 이슬; 이동욱; 정대웅
Original assignee: 주식회사 노타
Priority date: 2016-01-27
Filing date: 2016-01-27
Publication date: 2017-08-03
Also published as: KR20180105643A; KR102122438B1

Abstract

Disclosed are a display device and a touch input processing method therefor. A touch input processing method, according to an embodiment of the present invention, comprises the steps of: storing touch input data for a keyboard interface; generating at least one Gaussian model for each of the at least one key region included in the keyboard interface; generating a Gaussian mixture model for the keyboard interface by using the at least one Gaussian model; and mapping a received touch input as a key value by using the Gaussian mixture model.

Description

Display device and its touch input processing method

The present invention relates to a display device and a touch input processing method thereof, and more particularly, to generate a probability model by analyzing touch input data on a displayed soft keyboard interface, and to generate a user's touch input according to the generated probability model. It's about how to map the keys to your intentions.

Due to the development of display panel manufacturing technology and electronic device manufacturing technology, display devices of various sizes are becoming popular. Such devices may include a touch sensor to recognize a user's touch on the display panel and perform an operation accordingly. Recently, various portable devices such as mobile phones, notebook computers, PDAs, tablet PCs, and the like provide such a touch recognition function.

Touch-sensitive display devices can display a soft keyboard interface for user's character input. However, a typing error occurs due to a user's touch input method, a hand shape, a user's posture, a size of a display panel, and the like. Accordingly, touch input error correction methods based on language models have been developed. For example, the device may predict the character to be input according to the input characters, and correct the touch input error accordingly.

The error correction method according to language analysis / prediction requires large size data for various words, which are combinations of characters, and has a weakness in incompatibility because different models are required for each language. Therefore, the present specification is to propose a method that can reduce the typing error of the user while requiring less language-dependent constraints.

In order to solve the above technical problem, the touch input processing method according to an embodiment of the present invention, the step of storing the touch input data for the keyboard interface; Generating at least one Gaussian model for each of at least one key region included in the keyboard interface; Generating a Gaussian mixture model for the keyboard interface using the at least one Gaussian model; And mapping a received touch input to a key value using the Gaussian mixture model.

In addition, in the touch input processing method according to an embodiment of the present invention, the generating of the Gaussian model is performed using first touch input data included in a base area in the key area, and the base area is It can be a predefined area in the center of the key area.

In the touch input processing method according to an embodiment of the present invention, the generating of the Gaussian mixture model may include: first touch input data included in the base area and second touch input data not included in the base area; Can be performed using.

Further, in the touch input processing method according to an embodiment of the present invention, in the mapping of the received touch input to the key value, the received touch input is a first key of a first key region in which the received touch input is located. And a second key value of a second key area adjacent to the first key area.

Also, in the touch input processing method according to the embodiment of the present invention, the mapping of the received touch input to the key value may include: when the position of the received touch input is located within a high reliability area within the first key area. Mapping the received touch input to a first key value of the first key region including the high reliability region and if the received touch input is located outside the high reliability region within the first key region, the Gaussian mixing Mapping a constantly received touch input to a first key value of the first key area or to a second key value of a second key area adjacent to the first key area based on a model, wherein the high reliability The area may be a predefined area centered in the existing area.

In the touch input processing method according to the embodiment of the present invention, the mapping of the received touch input based on the Gaussian mixed model may include a preset value of the likelihood of the received touch input. Mapping the received touch input to a key value of the Gaussian mixed model when the threshold value is greater than or equal to a threshold value; The method may further include mapping a received touch input to the first key value of the first key area.

In addition, in the touch input processing method according to an embodiment of the present invention, the touch input data may be stored by randomizing the input order.

In addition, in the touch input processing method according to an embodiment of the present invention, if the number of the first touch input data is less than a predetermined number, deleting touch input data included in the key area including the base area. It may further include.

Further, in the touch input processing method according to the embodiment of the present invention, at least one of the size, shape, and position of at least one of the base area and the high reliability area may be variable.

In addition, the touch input processing method according to the embodiment of the present invention may display a GUI indicating a key mapping area reflecting the high reliability area and the Gaussian mixed model area together with the keyboard interface.

In addition, a display device according to an embodiment of the present invention for solving the above technical problem, a display unit for displaying a keyboard interface; A sensor unit for sensing a touch input to the display unit; A memory unit for storing touch input data for the keyboard interface and an application for processing the touch input; And a processing unit configured to drive the application to process the touch input, the display device generating at least one Gaussian model for each of the at least one key region included in the keyboard interface, One Gaussian model may be used to generate a Gaussian mixture model for the keyboard interface, and the Gaussian mixture model may be used to map a received touch input to a key value.

The display device according to the embodiment of the present invention may perform the touch input processing method according to the embodiment of the present invention. The display device according to the embodiment of the present invention may store a program / application for performing this touch input processing method. The display device according to an embodiment of the present invention may perform the touch input processing method of the present invention by driving such a program / application.

According to an embodiment of the present invention, a typo may be effectively corrected when a user inputs a touch. In particular, the present invention mathematically models the touch distribution and uses the distribution model to infer user intentions to map the received touch input to key values. Therefore, the present invention does not depend on the language of the keyboard interface, and thus can be applied to keyboard interfaces for various languages more universally. In addition, since the present invention does not depend on a language, a typo may be corrected by one modeling of one keyboard shape, and a plurality of models may not be required on one keyboard shape.

In modeling the touch distribution, the touch distribution for the base region is first modeled, and the entire touch distribution is modeled using the parameters obtained in the modeling. Therefore, the touch distribution can be modeled according to the user's intention. In particular, when the user's touch input is biased in a specific direction with a pattern, this can be effectively modeled.

The present invention can correct a typo by applying a modeled result and key mapping a touch input. However, even in this case, by using the high reliability region together with the modeling result, it is possible to prevent the correction of the mistake due to the excessive typo correction and the excessive typo correction by the modeling.

The present invention modifies a touch input, and provides a UI that can notify the distribution, statistics and modification of the touch input according to this process. Since the user can know his / her touch distribution or typo correction through the UI, the user can more easily recognize the usability and the use effect of the method / device / application / software according to the present invention.

In addition, the effect on the present invention will be further described in the following description.

1 illustrates a display device and a soft keyboard interface provided by the display device according to an embodiment of the present invention.

2 illustrates touch input data and error occurrences for the keyboard interface.

3 illustrates two key areas among key areas included in a keyboard interface.

4 illustrates user input for two key areas and two key areas included in a keyboard interface.

5 illustrates a method of initializing touch input data according to an embodiment of the present invention.

6 illustrates a Gaussian model generation method for each key region according to an embodiment of the present invention.

7 illustrates a Gaussian mixture model generation method for a keyboard interface according to an embodiment of the present invention.

8 illustrates a GMM region determination method according to an embodiment of the present invention.

9 illustrates a mapping method of a touch input using a GMM and a high reliability region according to an embodiment of the present invention.

10 illustrates a mapping method of a touch input using a GMM and a high reliability region according to an embodiment of the present invention.

11 illustrates a key mapping method of a received touch input according to an embodiment of the present invention.

12 illustrates a display device according to an embodiment of the invention.

13 is a flowchart illustrating a touch input processing method according to an embodiment of the present invention.

14 illustrates a graphical user interface (GUI) provided by a device or an application according to an embodiment of the present invention.

15 illustrates a keyboard interface and a calibrated keyboard interface in accordance with an embodiment of the present invention.

The terminology used herein is a general term that has been widely used as far as possible in consideration of the functions in the present specification, but may vary according to the intention of a person skilled in the art, convention or the emergence of a new technology. In addition, in certain cases, there is a term arbitrarily selected by the applicant, and in this case, the meaning will be described in the description of the corresponding embodiment. Therefore, it is to be understood that the terminology used herein is to be interpreted based not on the name of the term but on the actual meaning and contents throughout the present specification.

The present disclosure relates to a display device and a touch input processing method of the display device. The display device is meant to include a variety of electronic devices, for example, mobile phones, personal digital assistants (PDAs), notebooks, tablet PCs, MP3 players, CD players, DVD players, head mounted displays (HMDs), smart watches Various electronic devices capable of displaying visual information, such as a watch, a watch phone, a television, a kiosk, and a TV, are included. Hereinafter, for convenience of description, the display device may be referred to as a 'device'.

The invention will now be described with reference to a device for carrying out the method of the invention. However, the present invention can be implemented and performed by an application or software. In this case, the device is a device for driving an application or software for implementing / implementing the present invention, and the device is to implement the present invention according to the driving of the application or software. Thus, although the following describes the description of the invention with the aid of a device, the description of the invention should be considered to serve as a description of the application / software implementing the invention. The description of the method of the following specification and claims should be considered as describing the operation of the application / software in accordance with an embodiment of the invention. Thus, the following description of the present invention naturally applies to software / applications coded to perform the method of the present invention separately from the device. The operations of the devices described below may all be understood as operations of the software / application.

1 illustrates a display device 1010 according to an embodiment of the invention. Although the display device 1010 is shown as an embodiment of a tablet PC, the display device 1010 of the present invention is defined by the configuration and operation of the device, not the type or shape of the device.

The device 1010 may display the soft keyboard interface 1020. Soft keyboard interface 1020 represents a keyboard interface that includes at least one key region provided by being displayed without a physical mechanical configuration. The soft keyboard interface 1020 may be referred to herein as a virtual keyboard interface, a keyboard interface, or a soft keyboard.

The key area means an area corresponding to one displayed key. However, instead of the displayed area, an area recognized as a corresponding key value may be represented as a key area. In an embodiment, even when the keys of the displayed keyboard interface are spaced from each other, that is, even if there is a gap between the displanned keys, the reference area for recognizing the key value may not coincide with this gap. Therefore, in the present specification, the key area may indicate a displayed key layout or may indicate a default recognition area recognized by the corresponding key.

The keyboard interface 1020 may be provided in various forms. In FIG. 1, a keyboard interface 1020 having a key arrangement generally provided is illustrated as an example, but the shape of the keyboard interface 1020, the number of keys included, the arrangement of keys, the layout of the keyboard, and the like may vary according to embodiments. Can be. In addition, although FIG. 1 illustrates an embodiment in which the keyboard interface 1020 is displayed at the bottom of the display, the display position, orientation, and size of the keyboard interface 1020 are not limited thereto. The keyboard interface 1020 includes a plurality of key areas. And basically, the device maps touch inputs for key areas to the corresponding key values. For example, a touch input with coordinates in the key region of A is mapped to a key value of A.

When the keyboard interface is displayed as shown in FIG. 1, the user applies an input using a body part or an input tool to the display panel on which the keyboard interface is displayed. As means for sensing user input to a display, the device may include a touch sensor. The user input recognition method for the display of the touch sensor is a well-known technique and will not be described in detail.

The touch input may be dataized by the predetermined protocol of the touch sensor and the processor. That is, when the user applies a touch input to the display, the device may recognize the touch input as a touch sensor, and the recognized touch input may be processed as touch input data. The touch input data may include coordinate values (x, y). In an embodiment, the touch input data may further include touch start coordinates, touch end coordinates, touch input time, touch duration, touch area, additional sensor information of the touch time, environment information of the device, and the like. In an embodiment, the touch input data may include additional sensor information and may include tilt information and acceleration information of the device when the corresponding touch input is stored.

As in FIG. 2, the displayed keyboard interface includes a plurality of key areas. In the present specification, the key area may indicate an area corresponding to one displayed key. The device may recognize the touch input by mapping the touch input to the key value of the key area including the touch input coordinates. For example, when the location of the received touch input is located in the key region 2010 of 'ㅕ', the device may map the received touch input to a key value of 'ㅕ'. However, as shown in FIG. 2, an input for a key area not intended by the user may occur at the boundary of the key areas.

The dots in FIG. 2 represent stored touch input data. That is, the coordinate values of the touch input data are displayed as dots. In FIG. 2, the touch inputs marked with X are touch inputs recognized near the boundary of the key areas. As shown, in the case of specific keys, it can be seen that the distribution of touch input for the keys is biased in a specific direction. In FIG. 2, it can be seen that touch inputs to the key region 2010 of 'ㅕ' are biased to the right. Therefore, for the key area 2020 of the right 'ㅑ' key area 2010, the touch inputs of the left boundary of the key area 2020 correspond to the user's intention to view the touch input of the 'ㅕ' as the touch input. . Accordingly, the touch inputs of the left boundary of the key region 2020 of the 'ㅑ' may be determined as a typo of the user, and the typos may be corrected by mapping them to the key value of the 'ㅕ' rather than the key value of the 'ㅑ'.

The present invention proposes a method of mapping touch inputs to the boundary of the key areas to key values suitable for the user's intention through mathematical modeling. Hereinafter, a method of mapping a touch input for a key region of the keyboard interface to a key value according to a user's intention will be described in detail.

3 illustrates two key areas among key areas included in a keyboard interface.

3 illustrates a first key area 3010 for inputting a character A and a second key area 3020 for inputting a character S. Referring to FIG. The characters A and S are selected as embodiments, and the first key region 3010 and the second key region 3020 may be mapped to key values corresponding to arbitrary numbers and characters according to the configuration of the keyboard interface.

4 illustrates touch input data for two key areas and two key areas included in a keyboard interface.

As in FIG. 4, user input data for the keyboard interface may be collected. The user input data may include touch coordinates. The device may store more than a preset number of touch input data for a preset period of time, and perform touch input processing by analyzing and modeling the stored touch input data. The analysis and modeling of such data may be set to be performed at a time when the user does not use the device or at a time when the device is connected to the charger.

The device may perform data analysis and modeling when the touch input data for the entire keyboard interface is greater than or equal to a predetermined number. For example, when 3000 or more touch coordinate values of the touch input data are collected, the device may perform data analysis and modeling. However, in this case, the total number of data is 3000 or more, but the number of touch coordinates may be biased in a specific key area. Therefore, the device of the present invention can consider not only the number of global coordinate values but also the number of data for each key region. For example, when the number of recently received touch inputs for the keyboard interface is 3000 or more and the number of received touch inputs for each key area is 25 or more, the device may perform data analysis and modeling.

When the device stores touch input data, the device may randomize the storage order. The present invention performs touch input processing by analyzing / modeling only touch positions without performing language based touch input guessing. Therefore, even if the touch input data is not stored in order, it does not affect the analysis / modeling performance of the data. By randomizing the storage order, exposure of personal information such as passwords and specific word sequences can be minimized. In addition, when storing touch input data, an input to a personal information input interface such as a password field may not be stored. Touch input data is used locally only by the device and is not transmitted externally. However, if necessary, the touch input data may be transmitted to a server linked with the device. In this case, the server may be a server in which a program that manages / controls an app driven to implement the present invention in a device is stored and driven.

As shown in FIG. 4, the user input for the first key area 3010 and the second key area 3020 and the user input for S may have a distribution biased to the right. In this case, the touch inputs 4010 that are adjacent to the left side of the second key area 3020, that is, the boundary of the first key area 3010 are not the inputs to the second key area 3020, but the first key area 3010. Viewing as input to is consistent with the user's intent. Thus the mapping of inputs to the boundaries of these adjacent key regions can have a decisive effect on reducing typing errors. Hereinafter, a method of mapping such key inputs to key values suitable for user intention will be described in detail.

First, the present invention sets the

base areas

5010 and 5020 in the

key areas

3010 and 3020 of the keyboard interface, and starts modeling the touch input data in the base area. The device may map touch input data in the base area to key values of the corresponding area, and this mapping may be referred to as labeling. Labeling is the act of assigning a correct answer, which means mapping the received input to the correct result. In the present invention, the operation of mapping the received touch input to a correct key value may correspond to labeling. In the embodiment of FIG. 5, the device may map touch inputs in the first base area 5010 to a key value of 'A' and map touch inputs in the second base area 5020 to a key value of 'S'. Can be.

The first base area 5010 is positioned at the center of the displayed first key area 3010 and may have an area of a certain percentage of the key area. In an embodiment, the first base area 5010 with respect to the first key area 3010 occupies 80% of the area of the first key area 3010, and the shape is the same as that of the first key area 3010. It can be set to a shape. In an embodiment, the area, shape, and position of each base area with respect to each key area may be the same, but may be different.

The reason for using the base area is as follows. In FIG. 5, when modeling all touch inputs included in the key area to model the second key area 3020, the touch inputs 4010 intended for the input of the first key area are the second key area 3020. Modeled as input to. Therefore, since it is difficult to perform key value mapping according to user intention to the boundary region, the present invention first performs modeling based on the touch input of the base region, and then performs additional modeling using touch inputs other than the base region. Suggest a method.

The base area represents an area that can be estimated with a high probability that the key area will not be out of the user's intention even if it is recognized as a touch input to the corresponding key area.

In an embodiment, the device may delete the touch input data when the number of touch input data included in the base area in each key area is equal to or less than a preset number. That is, if the number of touch input data included in the base area is less than the predetermined number, the number of samples for proper Gaussian modeling is small, and thus Gaussian modeling for the corresponding key area may not be performed. In this case, if only the touch input data of the base area is deleted, the touch data remaining in the key area may affect the generation of the GMM for the entire keyboard interface. Therefore, when the number of touch input data for the base area is less than the preset number, the device may delete data for the key area as well as the base area, and generate the GMM using the deleted data.

As shown in FIG. 6, the present invention maps the touch input data 6010 in the first base area 5010 of the first key area 3010 to 'A'. That is, the touch input data 6010 in the first base area 5010 is mapped to a key value corresponding to the first key area 3010. In addition, the present invention can learn a Gaussian distribution using the mapped data. Training of the Gaussian distribution may be referred to as Gaussian model generation.

Gaussian probabilistic models are probabilistic models that represent a distribution of aggregated observations around a mean. The mathematical description of the Gaussian model itself is not detailed.

The present invention can generate a Gaussian model for each of the key areas included in the keyboard interface. For example, after the Gaussian distribution learning for the first key region, the present invention learns the second Gaussian distribution using the touch input data 6020 in the second base region 5020 of the second key region 3020. This operation can be performed for all key areas included in the keyboard interface. However, Gaussian modeling may be skipped for key regions where the number of sample data is less than a certain number.

Gaussian distribution can be learned / generated based on Maximum Likelihood. The learned parameters may be used as parameters of the GMM to be described later.

As shown in FIG. 6, the present invention learns a Gaussian distribution of individual key regions based on the base region, so that the Gaussian distribution of the first region 3010 and the Gaussian distribution of the second region 3020 are shifted to the right, respectively. You can expect it. That is, the mean of the Gaussian distribution (model) of the first region 3010 may be modeled to be close to the position of the touch input data 6010 within the first base region 5010. Similarly, the mean of the Gaussian distribution (model) of the second region 3020 will be modeled close to the location of the touch input data 6020 within the second base region 5020.

6, the present invention now generates a Gaussian Mixture Model (GMM) for the entire keyboard interface. The present invention models the GMM using all of the touch input data not used in the process up to FIG. 6 as shown in FIG. 7. In this case, the present invention can model the GMM using an Expectation Maximization (EM) technique. The GMM may indicate the probability of which key value the touch input for each key region of the keyboard interface should be mapped to.

Gaussian probabilistic models are probabilistic models that represent a distribution of aggregated observations around a mean. Therefore, there is a limitation that can express only a unimodal form in which data is grouped into a group centered on an average. Thus, the present invention uses GMM to represent a plurality of probability distributions for a plurality of keys included in a keyboard interface.

Expectation maximization technique is a method used to estimate the probability model with the received random variable. That is, the EM algorithm is an iterative algorithm that finds parameters with maximum likelihood or maximum a posteriori (MAP) in a probability model that depends on unobserved latent variables. The EM algorithm alternates between an expectation (E) step of calculating the expected value of log likelihood as an estimate of a parameter, and a maximization (M) step of obtaining a variable value that maximizes this expectation. The variable value calculated in the maximization phase is then used as an estimate for the next expected value phase.

The GMM is generated using a plurality of Gaussian models for each key region generated in the process up to FIG. 6. As an example, the GMM includes parameters of weight, mean, and covariance, wherein the double weight, mean, uses the parameters obtained for each Gaussian distribution, and the covariance is 1 / n. Can be initialized. n represents the number of keys modeled by GMM.

However, in the present invention, the parameter is optimized by repeatedly performing an EM technique in the modeling process of the GMM, and a value of the parameter may be out of an appropriate range and may be optimized to a local minimum value. For example, the position of the average of the Gaussian model with respect to region A in FIG. 7 may invade the key region of S, or the covariance may be too wide. Thus, the present invention may limit the width at which the mean and covariance can vary for each key region. For example, the present invention may limit an area in which an average of the Gaussian model for each key area may exist to an internal area such as the base area described above. It is also possible to set limits of the possible range of covariance.

Using the learned GMM, the likelihood of the Gaussian distribution with respect to the coordinates of the received touch input may be calculated. In addition, it is possible to estimate that the key value of the highest value is the user's intention by comparing the probability of all Gaussian distributions with respect to one coordinate. However, instead of comparing the probabilities of Gaussian distributions for all keys in order to reduce the amount of computation, the present invention may perform the comparison of the probabilities only for the input key and the adjacent keys. As an example of a typical alphabetic keyboard layout, the present invention provides a key in which a user input is located, such as performing a communicative comparison for S, A, and D, or S, A, D, W, and X for an input to S. You can also perform a communicative comparison on a region and two to eight adjacent key regions.

Hereinafter, a method of mapping a received touch input to a key value by comparing the probabilities of a plurality of Gaussian distributions included in the GMM will be described in more detail.

Fig. 8 shows a method for determining a GMM area by taking three key areas A, S, and D as an example. The three layout areas 8010 of FIG. 8A represent three key areas of the displayed keyboard interface. 8 shows a GMM for three key regions 8010 and a method for configuring the GMM regions accordingly.

8 (b) shows a GMM for three key areas A, S, and D. FIG. The GMM contains Gaussian distributions for each key region. In FIG. 8B, the GMM includes a Gaussian distribution 8020 for A, a Gaussian distribution 8030 for S, and a Gaussian distribution 8040 for D. In FIG. The present invention can map the coordinates to the key values of the Gaussian model if the probability of a particular coordinate is greater than or equal to a predetermined threshold based on each Gaussian model of the GMM.

In the embodiment of FIG. 8, it may be assumed that the coordinates of the specific touch input 8050 correspond to the position of x. The x position has the highest probable value for the Gaussian model 8040 of D in the GMM, which is above the threshold that can map the x coordinate to the key value of D. Therefore, although the device exists in the key area of S of the key area 8010 where the touch input 8050 is displayed, the device may map the touch input 8050 to a key value of D instead of S.

As shown in FIG. 8 (c), the present invention may configure a GMM region by collecting coordinates whose communicative value is greater than or equal to a threshold value from the GMM. In FIG. 8C, the GMM region includes a first region 8060 mapped to a key value of A, a second region 8070 mapped to a key value of S, and a third region 8080 mapped to a key value of D. It includes.

The threshold value can be determined / adjusted to reflect the user's intent. For example, increasing the threshold reduces the size of the GMM area where typos can be corrected, and can only correct typos for touch inputs that are close to the average of each Gaussian model. Therefore, the accuracy of the typo correction can be increased, but the frequency of typo correction can be reduced. Lowering the threshold can correct typos for touch inputs that are relatively far from the boundary. Instead, there is a risk that the area can be recognized as a typo by the intended touch input. Therefore, the accuracy of typos can be lowered, but the frequency of typos can be increased. The present invention is characterized by setting an appropriate threshold value in consideration of such a tradeoff.

The present invention is intended to correct typing errors that are not intended by the user, and determining and correcting the intended typing as an error is the most avoiding factor. Therefore, in addition to performing keymapping or thresholding only with GMM, a high-reliability zone can be set for a key zone, and touch input to this zone can always be mapped to key values for that zone. have.

FIG. 9 (a) shows three key areas of the keyboard interface displayed as shown in FIG. 8 (a).

9 (b) further shows a high reliability region within each key region. The high reliability area refers to an area that should be determined as a key input to the display area when a touch input is received in the area. As shown in FIG. 9B, the touch input in which the coordinate value is recognized into the high reliability region 9010 of the A key region is recognized as the A key regardless of the GMM.

FIG. 9C is a diagram illustrating the GMM region described above with reference to FIG. 8, and overlapping descriptions thereof will be omitted.

9 (d) shows final key mapping regions for mapping a touch input to a key value. As shown in FIG. 9 (d), the high reliability region is mapped to the corresponding key value irrespective of the GMM, and the regions of the threshold or higher in the GMM are set in addition to the high reliability region. The high reliability region and the remaining regions of the GMM region may be mapped according to the key values of the displayed key region. The finally determined regions may be referred to as key mapping regions. The key mapping area of A, the key mapping area of S, and the key mapping area of D are determined as shown in FIG. 9 (d). The entire keyboard interface including the plurality of key mapping regions may be referred to as a calibrated key mapping interface.

The high reliability region is located in the central region within the key region, and the size can be changed according to the setting. For example, when the user's touch input pattern, that is, the average value of the GMMs, is largely out of the center of each key region, the size of the high reliability region may be reduced to increase the error correction capability. Alternatively, when the user's touch input pattern is generally located close to the center of each key region, the size of the high reliability region may be increased to reduce incorrect typographical correction and to improve typo correcting accuracy. The location of the high reliability region may also be adjusted according to the user's touch input pattern to improve the error correction capability and accuracy. For example, in the key region of 'D' of FIG. 9 according to the user's touch input pattern, the high reliability region may be moved to the left to improve both error correction capability and accuracy.

FIG. 10 shows the method shown in FIG. 9 with respect to the x plane.

Fig. 10 (a) shows the displayed layout area of Fig. 9 (a), Fig. 10 (b) shows the high reliability area of Fig. 9 (b), and Fig. 10 (c) shows the GMM area of Fig. 9 (c). 10 (d) and 9 (d) show the final determined key mapping inverse in the x plane, respectively.

As shown in FIG. 10, in determining the final region, the present invention considers the GMM region but does not modify the high reliability region. That is, the present invention configures the final key mapping region so that the GMM region for 'D' of FIG. 10 (c) does not exceed the high reliability region for 'S' of FIG. 10 (b).

In addition, the present invention configures a key mapping area according to the GMM area for an area greater than or equal to a specific threshold in the GMM, and follows the mapping of the displayed key area for the high reliability area and the outside of the GMM area.

As described above with reference to FIGS. 8 to 10, the present invention may configure a key mapping area for the entire keyboard interface, and perform key mapping according to a position in the key mapping area of the coordinates of the received touch input. However, in this embodiment, the amount of pre-computation may increase. Therefore, as another embodiment, the present invention may process the received touch input for each coordinate without configuring the entire key mapping area. 11 is a flow chart illustrating this method.

The device may receive a touch input for the displayed keyboard interface in operation S11010. The received touch input includes coordinate information.

The device may first determine whether the coordinates of the received touch input are located in the high reliability area (S11020). If the touch input coordinates exist in the high reliability region, the device may map the received touch input to a key value of the high reliability region (S11030).

If the coordinates of the received touch input are not located in the high reliability region, the device may determine whether the GMM corresponding to the coordinates of the touch input is equal to or greater than the threshold (S11040). If the GMM corresponding to the touch input coordinates is equal to or larger than the threshold, the device may map the received touch input to a key value of the corresponding GMM region (S11050). In other words, the device may map the received touch input to a key value of the Gaussian model when the communicability of the received touch input coordinates is equal to or larger than a threshold value for one of the plurality of Gaussian models included in the GMM. have.

When the coordinates of the received touch input are outside the high reliability region and less than the communicability threshold of the GMM, the device may map the touch input to a key value of the key region where the touch input is located (S11060).

12 illustrates a display device according to an embodiment of the invention.

In FIG. 12, the display device 12000 may include a display unit 12010, a communication unit 12020, a processing unit 12030, a sensor unit 12040, and a memory unit 1250. As indicated by the dotted lines in FIG. 12, the communication unit 12020 may optionally be included.

The display unit 12010 may display visual information on the display screen. Here, the visual information may represent a still image, a moving picture, an application execution screen, various interfaces, or various visually expressible information including the same, which can be displayed by the display unit 1200. The display unit 12010 may output various visual information on the display screen based on the control command of the processing unit 12030. In particular, the display unit 12010 of the present invention can display a soft keyboard interface as shown in FIG. 1, and display GUIs according to other embodiments of the present invention.

The communication unit 12020 may perform wired or wireless communication with an external device of the device. The communication unit 12020 may not be provided depending on the configuration of the device, or may be configured of a plurality of communication chipsets. The communication unit 1230 includes a communication module and may perform communication using various communication protocols such as 3G, 4G (LTE), 5G, WIFI, Bluetooth, and NFC.

The sensor unit 12040 may sense a user input or an environment of the device by using at least one sensor mounted on the device. In an embodiment, the at least one sensor is a variety of sensors, such as touch sensors, fingerprint sensors, motion sensors, pressure sensors, camera (image) sensors, tilt sensors, gyro sensors, gyroscope sensors, angular velocity sensors, illuminance sensors, and angle sensors. It may include. The above-described sensors may be included in the device as a separate element, or may be integrated into at least one or more elements and included in the device. In particular, the sensor unit 12040 of the present invention includes a touch sensor capable of sensing a touch input to the display unit 12010. The touch sensor may sense a touch input and transmit data of a promised type, such as a coordinate value, to the processing unit 1230. Here, the touch input includes both contact and non-contact touch inputs (eg, hovering inputs) to the display unit 12010. In addition, the touch input may contact or not touch the display unit 12010 using touch input means (for example, a stylus pen or a touch pen), as well as an input for directly contacting or non-contacting the display unit 12010 with a part of a user's body. It includes all of the touch input.

The memory unit 1250 may store data including various information. The memory unit 12050 collectively refers to volatile and nonvolatile memories. In the present invention, the memory unit 1250 may store the received touch input data. In addition, in the present invention, the memory unit may store an application / software for driving the above-described method of the present invention.

The processing unit 1230 may control at least one other unit included in the device. The control unit 1230 may process data inside the device. In addition, the control unit 1230 may control at least one unit included in the device based on the detected user input. Processing unit 12030 may implement the method of the present invention by running an application / software for performing the method of the present invention.

As described above, the touch input processing method according to the present invention may be performed by a device or by an application / software running on the device.

The device may store touch input data for the keyboard interface in operation S13010. The displayed keyboard interface may include at least one key area, which may be referred to as a layout area. The touch input data may include at least one of coordinates of the touch input, an area of the touch input, a duration, and additional sensor information at the time of the touch input. The device may randomly store the storage order of the touch input data.

The device may generate a Gaussian model for each key region (S13020). The device may generate a Gaussian model / distribution using the stored touch input data. The device may generate a Gaussian model using only touch input data in the base area of each key area. The device may generate a Gaussian model for all or some of the key areas included in the keyboard interface.

In an embodiment, when the number of key areas included in the keyboard interface is n, the device may generate up to n Gaussian models. As described above, the Gaussian model generation for the m key areas may be omitted according to the number of touch input data. In this case, the device may generate n-m Gaussian models and generate a GMM for the keyboard interface. The device may map the touch input according to the displayed key area for the key area in which the Gaussian model is not generated.

The device may generate a Gaussian mixture model (GMM) for the keyboard interface using the generated Gaussian model (S13030). The device may generate a Gaussian mixture model by using touch input data in the base area and touch input data other than the base area together. The description of FIGS. 3 to 8 applies to a Gaussian model generation of the device and a method of generating a Gaussian mixed model using the same, and overlapping descriptions will not be repeated.

The device may map the received touch input to a key value using a Gaussian mixed model (S13040). The device may map the received touch input to a key value of a key area where the received touch input is located or a key value of an adjacent key area of the key area where the received touch input is located. The key value may represent a character, a number, a symbol, or the like displayed on the keyboard interface, or a digital value corresponding to the character, number, symbol, or the like.

The device may perform key value mapping of the received touch input as described with reference to FIGS. 9 through 11. The device may map the touch input based on the key mapping area reflecting the high reliability area and the GMM area. Alternatively, the device may map the touch input based on the coordinate of the touch input as in the method of FIG. 11.

The device of the present invention is characterized in that a touch input is mapped to a key value of a key region where a touch input is located or a key mapping region where a touch input is located to a key value of an adjacent key region. That is, the input of a specific key region may be mapped to the key value of an adjacent key region according to the distribution of the received touch input without changing the layout of the displayed key regions.

The device may first determine whether the location of the received touch input is a high reliability area of a particular key area. If the received touch input is located in the high reliability region, the device may map the received touch input to a key value of a key region including the corresponding high reliability region. When the received touch input is located outside the high reliability region, the device may map the received touch input to a key value of a key region where the received touch input is located or a key value of an adjacent key region based on the Gaussian mixture model.

The method of mapping the received touch input based on the mixed Gaussian model is as described with reference to FIGS. 8 to 11. When the communicative value of the touch input is equal to or greater than the threshold value of the GMM, the device may map the received touch input to a key value of the GMM that is greater than or equal to the threshold value. The key value of the GMM that is greater than or equal to the threshold value may be the key value of the Gaussian model whose communicative value of the corresponding coordinate is greater than or equal to the threshold value among the plurality of Gaussian models represented by the GMM. When the communicative value of the received touch input is less than a predetermined threshold value of the mixed Gaussian model, the device may map the received touch input to a key value of a key region in which the received touch input is located.

The device may randomly store an input order of touch input data. In addition, when the number of first touch input data included in the base area is less than the preset number, the touch input data included in the key area including the base area may be deleted. Therefore, Gaussian modeling of the corresponding key region is not performed, and touch data of the corresponding key region does not affect the generation of the GMM for the keyboard interface.

As described above, the size, shape, position, etc. of at least one of the base region and the high reliability region may be variable. According to the embodiment, the base region and the high reliability region may be set identically.

The following describes the GUI that can be additionally displayed as an embodiment according to the present invention.

When the touch input is processed using the above-described method, a UI (User Interface) for informing the user of such processing may be provided. For example, when correcting a typo according to the above-described method, the device may provide an effect of notifying the user of the correction of the typo. This effect may be provided as a tactile, sound or visual effect such as vibration.

14 illustrates an embodiment of a GUI illustrating typo correction. When a typo is corrected by the touch input processing method of the present invention as shown in FIG. 14, the device may display the modified key area by coloring or highlighting it. Correcting a typo in this specification indicates a case where a received touch input is mapped to a key value of an adjacent key region instead of a key region in which a touch input is located according to a GMM region. That is, when a touch input is received in an area other than the key area of "ㅏ" as shown in FIG. 14 (b), but is mapped to a key value of "ㅏ", the device (app) has a visual effect in the key area of "ㅏ". Can be applied. FIG. 14C illustrates a visual effect when a touch input is normally received in a key region of “” and the touch input is mapped to a key value of “”.

The user can know when a typo was corrected by the present invention through the GUI as shown in FIG.

15 illustrates a recognition area 15010 of the keyboard interface before applying the present invention and a key mapping area 15020 of the corrected keyboard interface to which the present invention is applied. In addition, as shown in FIG. 15, the touch input data may be displayed as a dot to show a user input pattern. In particular, a visual effect may be added to and displayed on the touch input data determined to be a typo. In FIG. 15, touch input data recognized as a typo is denoted by x. However, according to an exemplary embodiment, the device (app) may display the data to identify data with different points. The GUI of the present invention may provide a distribution of key mapping areas and touch input data to the keyboard interface.

In an embodiment of the present invention, the device (app) may provide as a GUI the recognition area 15010 of the keyboard interface and the key mapping area 15020 of the calibration keyboard interface before applying the present invention, and provide statistics according to the calibration. It may be. In the embodiment of FIG. 15, when the present invention is applied, the error rate is reduced from 2.03% to 1.39%, the typing speed is increased from 163 CPM (Characters Per Minute) to 281 CPM, and the frequency of using a backspace is 100. It is reduced from 9.4 times to 5.1 times.

In an embodiment of the present invention, the device (app) may provide a calibrated key mapping interface 15020 to the user and also provide statistical data / tables / graphs according to the calibration. In an embodiment of the present invention, the device (app) may further include a separate key in the keyboard interface that can trigger the display of the present invention's additional GUI, typographical error, distribution of touch input data, and the like.

It is understood by those skilled in the art that various changes and modifications can be made in the present invention without departing from the spirit or scope of the invention. Accordingly, it is intended that the present invention cover the modifications and variations of this invention provided they come within the scope of the appended claims and their equivalents.

Reference is made herein to both apparatus and method inventions, and the descriptions of both apparatus and method inventions may be complementary to one another.

Various embodiments have been described in the best mode for carrying out the invention.

The present invention can be used in the field of touch recognition of display devices and display devices.

Claims

In the touch input processing method for the displayed keyboard interface,

Storing touch input data for the keyboard interface;

Generating at least one Gaussian model for each of at least one key region included in the keyboard interface;

Generating a Gaussian mixture model for the keyboard interface using the at least one Gaussian model; And

Mapping a received touch input to a key value using the Gaussian mixture model.
The method of claim 1,

Generating the Gaussian model;

And is performed using first touch input data included in a base area in the key area, wherein the base area is a predefined area centered in the key area.
The method of claim 2,

Generating the Gaussian mixture model,

And first touch input data included in the base area and second touch input data not included in the base area.
The method of claim 1,

In the mapping of the received touch input to the key value,

And wherein the received touch input is mapped to a first key value of a first key region in which the received touch input is located or a second key value of a second key region adjacent to the first key region.
The method of claim 1,

The mapping of the received touch input to the key value includes:

Mapping the received touch input to a first key value of the first key region including the high reliability region when the location of the received touch input is located within a high reliability region within a first key region;

When the received touch input is located outside the high reliability region in the first key region, a constantly received touch input is applied to the first key value or the first key region of the first key region based on the Gaussian mixture model. Mapping to the second key value of an adjacent second key region;

And wherein said high reliability region is a predefined region centered within said region.
The method of claim 5,

Mapping the received touch input based on the Gaussian mixture model,

Mapping the received touch input to a key value of the Gaussian mixture model when the likelihood value of the received touch input is greater than or equal to a predetermined threshold value of the Gaussian mixture model;

Mapping the received touch input to the first key value of the first key region if the received touch input is less than a preset threshold value of the Gaussian mixture model; , Touch input processing method.
The method of claim 1,

And the touch input data is stored by randomizing the order of input.
The method of claim 3, wherein

And deleting the touch input data included in the key area including the base area when the number of the first touch input data is less than a preset number.
The method according to claim 2 or de 5,

And at least one of the size, shape, and position of at least one of the base region and the high reliability region is variable.
The method of claim 4, wherein

And a GUI indicating a key mapping area reflecting the high reliability area and the Gaussian mixed model area together with the keyboard interface.
A display unit for displaying a keyboard interface;

A sensor unit for sensing a touch input to the display unit;

A memory unit for storing touch input data for the keyboard interface and an application for processing the touch input; And

A display device comprising a processing unit that drives the application to process touch input,

The display device,

Generate at least one Gaussian model for each of the at least one key region included in the keyboard interface, generate a Gaussian mixture model for the keyboard interface using the at least one Gaussian model, and generate the Gaussian mixture model Display to map incoming touch input to a key value.
The method of claim 11,

The generation of the Gaussian model is performed using first touch input data included in a base area in the key area, wherein the base area is a predefined area centered in the key area.
The method of claim 12,

The generation of the Gaussian mixed model is performed using first touch input data included in the base area and second touch input data not included in the base area.
The method of claim 11,

In the mapping of the received touch input to the key value,

And the received touch input is mapped to a first key value of a first key region in which the received touch input is located or a second key value of a second key region adjacent to the first key region.
The method of claim 12,

The mapping of the received touch input to the key value is

When the location of the received touch input is located in a high reliability area within the first key area, the received touch input is mapped to a first key value of the first key area including the high reliability area, and the received touch input When located outside the high reliability region in the first key region, a constantly received touch input is based on the Gaussian mixture model to generate the first key value of the first key region or a second key adjacent to the first key region. By mapping to the second key value of the region,

And the high reliability region is a predefined region centered in an existing region.
The method of claim 15,

The mapping of the received touch input based on the Gaussian mixture model is

When the likelihood value of the received touch input is equal to or greater than a preset threshold value of the Gaussian mixed model, the received touch input is mapped to a key value of the Gaussian mixed model, and the communicative value of the received touch input is Gaussian. And if the received touch input is mapped to the first key value of the first key region when the received touch input is less than a predetermined threshold value of a mixed model.
The method of claim 11,

And the touch input data is stored by randomizing the input order.
The method of claim 13,

And when the number of the first touch input data is less than a predetermined number, the touch input data included in the key area including the base area is deleted.
The method according to claim 12 or 15,

And at least one of the size, shaping and position of at least one of the base region and the high reliability region is variable.
The method of claim 14,

And a GUI indicating a key mapping area reflecting the high reliability area and the Gaussian mixed model area together with the keyboard interface.