WO2024234372A1

WO2024234372A1 - Methods and systems for pressure-assisted interaction with a touch sensitive input device

Info

Publication number: WO2024234372A1
Application number: PCT/CN2023/095067
Authority: WO
Inventors: Soheil Kianzad; Che YAN; Jay Henderson; Wei Li
Original assignee: Huawei Technologies Co., Ltd.
Priority date: 2023-05-18
Filing date: 2023-05-18
Publication date: 2024-11-21

Abstract

There are provided methods and devices for improving user interaction with an electronic device having a touch sensitive surface, for example, using multiple input modes. Specifically, user interaction in the form of stroke generation gestures on an electronic device may be assisted by input from pressure information. In response to detecting a stroke generation gesture, rich stroke information corresponding to the stroke generation gesture is received. In some examples, rich stroke information includes positional information generated by the touch sensitive surface and pressure information. Using the positional information and the pressure information, a pressure-gesture recognizer generates a pressure-assisted gesture for controlling a function of the electronic device. The disclosed method and device may enable improved interaction and/or character recognition for applications using strokes as input, for example, handwritten text, drawing or user interfaces, while overcoming challenges associated with accurately characterizing small or continuous strokes.

Description

METHODS AND SYSTEMS FOR PRESSURE-ASSISTED INTERACTION WITH A TOUCH SENSITIVE INPUT DEVICE

TECHNICAL FIELD

The present disclosure relates to the field of human-computer interaction, in particular, methods and systems for handwriting, character and stroke recognition for strokes generated on a touch screen display.

BACKGROUND

Handwriting is composed of a series of strokes that form straight lines and curved lines of various sizes and directions, that that are combined in a specific way to form letters, words, numbers, symbols or characters. Advances in digital stylus technologies and handwriting recognition has made handwriting more common among tablet and digital stylus users; however, handwriting recognition remains challenging, due to inherent differences between individual handwriting style.

Optical character recognition (OCR) is commonly used to convert images of typed, printed or handwritten text into machine-readable text. Due to variations in handwriting style, it can be difficult to accurately recognize characters in handwritten text. One example approach to improve handwritten character recognition is described in: El Bahi, Hassan, Zouhir Mahani, and Abdelkarim Zatni, "An offline handwritten character recognition system for image obtained by camera phone" Université Ibn Zohr, ESTA, Laboratoire Matériaux, Systèmes et Technologies de l’information, the entirety of which is hereby incorporated by reference. In Bahi et al., English character recognition using feature extraction and classification employed various methods such as grey level, co-occurrence matrix, Zernike moments, Gabor filters, zoning, projection histogram and distance profile for the purpose of feature extraction. However, the approach was limited to large, isolated characters and not cursive handwriting or other handwriting styles where individual letters or characters may be joined together.

An example approach to OCR for cursive handwriting is described in: Dhande, P., & Kharat, R. (2017, May) , Recognition of cursive English handwritten characters. In 2017 International Conference on Trends in Electronics and Informatics (ICEI) (pp. 199-203) , IEEE, the entirety of which is hereby incorporated by reference. A primary objective of the method in Dhande and Kharat, 2017 was to increase the speed of character recognition, and accuracy was not evaluated.

An example approach to handwritten character recognition using a combined vector field method that employs auto-correlation matrix and low frequency Fourier expansion is described in: Izumi, T., Hattori, T., Kitajima, H., & Yamasaki, T. (2004, October) , Characters recognition method based on vector field and simple linear regression model, In IEEE International Symposium on Communications and Information Technology, 2004, ISCIT 2004, (Vol. 1, pp. 498-503) , IEEE, the entirety of which is hereby incorporated by reference. The method in Izumi et al. provided good improvement over traditional character recognition methods.

Another example of OCR for cursive handwriting is described in: Arica, N., & Yarman-Vural, F.T. (2002) , Optical character recognition for cursive handwriting, IEEE transactions on pattern analysis and machine intelligence, 24 (6) , 801-813, the entirety of which is hereby incorporated by reference. Arica et al. proposed a sequence image segmentation and recognition algorithm for offline cursive handwriting recognition consisting of three steps: 1) Estimation of global parameters, such as slant angle, baselines, stroke width and height, 2) Combining gray scale and binary information to find character segmentation and 3) Employ a hidden Markov model (HMM) for shape recognition and rank the character candidates.

Another example of OCR for cursive handwriting is described in: Zhu, B., Shivram, A., Setlur, S., Govindaraju, V., & Nakagawa, M. (2013, August) , Online handwritten cursive word recognition using segmentation-free MRF in combination with P2DBMN-MQDF, In 2013 12th International Conference on Document Analysis and Recognition (pp. 349-353) . IEEE, the entirety of which is hereby incorporated by reference. Zhu et al. disclose a segmentation-free Markov random field (MRF) model in combination with an offline recognition method based on pseudo 2D bi-moment normalization (P2DBMN) which has been widely used for Chinese character recognition.

Accordingly, it would be useful to provide methods and systems to improve stroke and character recognition for handwritten text.

SUMMARY

In various examples, the present disclosure describes methods and devices for improved user interaction with an electronic device having a touch sensitive surface, for example, using multiple input modes. Specifically, user interaction in the form of stroke generation gestures on an electronic device may be assisted by input from pressure information. In response to detecting a stroke generation gesture, rich stroke information corresponding to the stroke generation gesture is received. In some examples, rich stroke information includes positional information generated by the touch sensitive surface and pressure information. Using the positional information and the pressure information, a pressure-gesture recognizer generates a pressure-assisted gesture for controlling a function of the electronic device. The disclosed methods and devices may enable improved UI interaction and/or character recognition for applications using strokes as input, for example, handwritten text, drawing or user interfaces, while overcoming challenges associated with accurately characterizing small or continuous strokes, for example, including punctuation or cursive writing.

In various examples, the present disclosure provides the technical effect that a pressure-assisted gesture is generated for improving stroke recognition and other stroke-based interactions on a touch sensitive surface of an electronic device. In examples, a pressure-assisted gesture may include a pressure-assisted tapping gesture or a shear-assisted stroke gesture, or a combination thereof, among others. In examples, controlling functions of the electronic device using pressure-assisted gestures may provide advantages in detecting small stroke gestures associated with punctuation that may normally be too small to be accurately characterized by an optical character recognition (OCR) application. In other examples, pressure-assisted gestures may provide advantages in segmenting characters in cursive handwriting or other handwriting styles where individual letters or characters may be joined together.

In an example aspect, the present disclosure describes a method for interacting with an electronic device having a touch sensitive surface configured to detect a stroke generation gesture on the touch sensitive surface. The method includes: receiving a rich stroke information corresponding to the stroke generation gesture, the rich stroke information including a positional information and a pressure information; computing, by a pressure-gesture recognizer using the positional information and the pressure information: a pressure-assisted gesture; and controlling a function of the electronic device based on the pressure-assisted gesture.

In an example of the preceding example aspect of the method, the touch sensitive surface is a capacitive touch sensitive surface and the stroke generation gesture represents a movement of a tip portion of a digital pen device in contact with the capacitive touch sensitive surface.

In an example of any one of the preceding example aspects of the method, the positional information comprises a position profile including set of coordinates corresponding to points of contact of the digital pen device on the touch sensitive surface in response to the stroke generation.

In an example of any one of the preceding example aspects of the method, the pressure information comprises a pressure profile describing a time-varying measure of a magnitude of pressure applied by the digital pen device to the touch sensitive surface in response to the stroke generation.

In an example of any one of the preceding example aspects of the method, the rich stroke information includes a force information comprising a first tangential force and a second tangential force applied by the digital pen device to the touch sensitive surface in response to the stroke generation gesture, and the pressure-assisted gesture is further computed based on the force information.

In an example of any one of the preceding example aspects of the method, the rich stroke information includes a pen motion information describing an orientation of the digital pen device in response to the stroke generation gesture, the motion information generated by a motion sensor coupled to the digital pen device, and the pressure-assisted gesture is further computed based on the motion information.

In an example of any one of the preceding example aspects of the method, generating the pressure-assisted gesture comprises: preprocessing the rich stroke information by resampling, rotating and scaling the positional information associated with the stroke generation gesture; inputting the preprocessed rich stroke information into a Euclidean-distance based stroke recognizer to compare the preprocessed rich stroke information to a plurality of stored templates; computing an average Euclidean distance between the preprocessed rich stroke information and each of the plurality of stored templates; and determining a best match template of the plurality of stored templates corresponding to a minimum average Euclidean distance.

In an example of any one of the preceding example aspects of the method, generating the pressure-assisted gesture comprises: preprocessing the rich stroke information using a dynamic time warping method to determine a weighted DTW distance associated with the stroke generation gesture; inputting the preprocessed rich stroke information into a Euclidean-distance based stroke recognizer to compare the preprocessed rich stroke information to a plurality of stored templates; computing an average Euclidean distance between the preprocessed rich stroke information and each of the plurality of stored templates; and determining a best match template of the plurality of stored templates corresponding to a minimum average Euclidean distance.

In an example of any one of the preceding example aspects of the method, generating the pressure-assisted gesture comprises: preprocessing the rich stroke information to obtain directional shear forces associated with the stroke generation gesture; inputting the preprocessed rich stroke information into a Euclidean-distance based stroke recognizer to compare the preprocessed rich stroke information to a plurality of stored templates; computing an average Euclidean distance between the preprocessed rich stroke information and each of the plurality of stored templates; and determining a best match template of the plurality of stored templates corresponding to a minimum average Euclidean distance.

In an example of any one of the preceding example aspects of the method, generating the pressure-assisted gesture comprises: computing, by a machine learning (ML) model using the rich stroke information, the pressure-assisted gesture.

In an example of the preceding example aspect of the method, training the ML model comprises: obtaining a training dataset comprising rich stroke information for a plurality of example stroke generation gestures; computing a pressure profile for each one of the plurality of example stroke generation gestures, based on the rich stroke information; and training the ML model based on the rich stroke information and the pressure profiles.

In an example aspect, the present disclosure describes a digital pen device for interacting with a touch sensitive surface of an electronic device. The digital pen device includes: a flexible tip portion comprising a metal electrode for contacting the touch sensitive surface; a processing unit configured to execute instructions to cause the device to: generate a rich stroke information corresponding to a stroke generation gesture, the rich stroke information including a positional information and a pressure information.

In an example of the preceding example aspect of the device, the touch sensitive surface is a capacitive touch sensitive surface and the stroke generation gesture represents a movement of the flexible tip portion in contact with the capacitive touch sensitive surface.

In an example of any one of the preceding example aspects of the device, the touch sensitive surface is a capacitive touch sensitive surface and the stroke generation gesture represents a movement of the flexible tip portion in contact with the capacitive touch sensitive surface.

In an example of any one of the preceding example aspects of the device, the rich stroke information includes a force information comprising a first tangential force and a second tangential force applied by the flexible tip portion to the touch sensitive surface in response to the stroke generation gesture.

In an example of the preceding example aspect of the device, the force information is measured based on a bending of the flexible tip portion in contact with the touchscreen display.

In an example of the preceding example aspect of the device, further comprising: a force sensor coupled to the flexible tip portion for receiving a force input applied to the flexible tip portion in response to the stroke generation gesture, wherein the rich stroke information includes a force information comprising a first tangential force and a second tangential force applied by the flexible tip portion to the touch sensitive surface in response to the stroke generation gesture.

In an example of the preceding example aspect of the device, further comprising: a motion sensor coupled to the digital pen device for receiving a motion input corresponding to a motion of the digital pen device in response to the stroke generation gesture; wherein the rich stroke information includes a motion information comprising an orientation of the digital pen device in response to the stroke generation gesture.

In an example aspect, the present disclosure describes a non-transitory computer-readable medium storing instructions executable by a processing unit of a computing system to cause the system to: receive a rich stroke information corresponding to a stroke generation gesture, the rich stroke information including a positional information and a pressure information; compute, by a pressure-gesture recognizer using the positional information and the pressure information: a pressure-assisted gesture; and control a function of the electronic device based on the pressure-assisted gesture.

BRIEF DESCRIPTION OF THE DRAWINGS

Reference will now be made, by way of example, to the accompanying drawings which show example embodiments of the present application, and in which:

FIG. 1 is a block diagram illustrating an example computing system which may be used to implement examples of the present disclosure;

FIG. 2 is a block diagram of an example pressure-assisted gesture recognition system, in accordance with examples of the present disclosure;

FIG. 3A illustrates a side view of an example embodiment of a digital pen device including a flexible tip in contact with a capacitive touch sensitive surface producing a corresponding capacitive image, in accordance with examples of the present disclosure.

FIG. 3B illustrates a side view of an example embodiment of the digital pen device including a flexible tip in contact with the capacitive touch sensitive surface producing a corresponding capacitive image, in accordance with examples of the present disclosure.

FIG. 3C illustrates a magnified image of an example embodiment of a tip electrode of the digital pen device, in accordance with examples of the present disclosure.

FIG. 3D illustrates an example approach for computing a pseudo shear vector for a digital pen device, in accordance with examples of the present disclosure.

FIG. 4 is a flowchart illustrating an example algorithm for stroke recognition using pressure-assisted gestures, in accordance with an example embodiment of the present disclosure;

FIG. 5A to 5C illustrate an example of handwriting where individual letters are joined, in accordance with an example embodiment of the present disclosure;

FIG. 6 is a flowchart illustrating an example algorithm for character segmentation using pressure-assisted gestures, in accordance with an example embodiment of the present disclosure.

FIG. 7 is a flowchart illustrating an example algorithm for recognizing connections between adjacent characters in handwritten text using pressure-assisted gestures, in accordance with an example embodiment of the present disclosure.

FIG. 8 is a flowchart illustrating an example algorithm for recommending potential characters in handwritten text in real-time using pressure-assisted gestures and stroke order, in accordance with an example embodiment of the present disclosure.

FIG. 9 is a schematic diagram illustrating an example algorithm for state transition between drag and shear, using pressure-assisted gestures, in accordance with an example embodiment of the present disclosure.

FIG. 10 is a flowchart illustrating an example algorithm for UI interaction and object selection, using pressure-assisted gestures, in accordance with an example embodiment of the present disclosure.

FIG. s 11A-F illustrate example embodiments of stationary pressure-assisted gestures for interacting with a user interface of an electronic device, in accordance with examples of the present disclosure.

FIG. 12 is a flowchart illustrating an example algorithm for enhanced personalization of stroke gestures using pressure-assisted gestures, in accordance with an example embodiment of the present disclosure.

FIG. 13 is a flowchart illustrating an example method for interacting with an electronic device having a touch sensitive surface, in accordance with examples of the present disclosure;

Similar reference numerals may have been used in different figures to denote similar components.

DETAILED DESCRIPTION

The following describes example technical solutions of this disclosure with reference to accompanying drawings. Similar reference numerals may have been used in different figures to denote similar components.

To assist in understanding the present disclosure, some existing challenges associated with OCR for handwritten text are discussed.

One drawback of current approaches to handwritten text recognition is inaccuracy in detecting punctuation. Conventional methods such as resampling and scaling, which are widely used in OCR technologies, are not appropriate for small strokes such as those associated with punctuation marks. For example, a small stroke has fewer data points (e.g., 5-20 points) than a large stroke, and when the small stroke is resampled and scaled, it can erroneously resemble another character. Moreover, accurately recognizing punctuation can also be challenging when punctuation overlaps with text.

Related to the challenges in punctuation recognition, it is understood that automatic speech recognition technologies lack effective tools for inserting punctuation. For example, Speech-to-Text tools do not include punctuation marks in the speech recognition output (e.g., text transcript) . In general, detecting the prosodic features of the speech and adding the punctuation to ASR outputs is a complex task. In examples, one approach to adding punctuation to ASR transcripts include post-processing with a natural language processor (NLP) to detect and insert punctuation in the transcript, however the accuracy of such post-processing approaches is suboptimal.

Another drawback of current approaches to handwritten text recognition is inaccuracy in accurately segmenting and recognizing characters that may be joined together, for example, as in cursive writing or other writing styles where individual characters are effectively connected. For example, consider a handwritten text including the letter “r” followed by the letter “n” , where the two letters are effectively connected. A typical OCR tool may mistake the input for the letter “m” . In situations where OCR cannot effectively recognize an individual character, the OCR tool may need to wait until the entire word (e.g., English) or the entire character (e.g., Chinese) is completed to recognize the content, which may slow down the input efficiency or limit the use of auto-complete suggestions based on partial word/character input (e.g., Pinyin) . In this regard, any errors in spelling or other typographical errors may further hinder OCR accuracy. Moreover, it is understood that interaction with digital devices using a stylus or digital pen provides a different user experience than writing with a pen and paper, for example, the surface of a touch screen surface may lack friction, or a user may require a different writing posture to avoid touching their wrist or palm on the touch sensitive surface, which may impact the quality of the handwriting and further hinder OCR.

In some embodiments, the present disclosure describes examples that address some or all of the above drawbacks of existing techniques for character recognition and segmentation of handwritten text.

To assist in understanding the present disclosure, the following describes some relevant terminology that may be related to examples disclosed herein.

In the present disclosure, “optical character recognition (OCR) ” can mean: the conversion of images of typed or handwritten text into machine-readable text, for example, from a scanned document, a photograph or a screenshot, among other images. OCR commonly employs pattern matching to compare a text image to stored templates representative of different fonts and text images, to detect each character. Alternatively, OCR may employ feature extraction approaches, for example, using trained machine learning models to classify individual characters.

In the present disclosure, “handwriting” can mean: any form of writing (e.g., generation of strokes in recognizable patterns) by an individual while holding a writing instrument, such as a pen, pencil, stylus, wand or mouse, among others, or while performing a touch gesture, such as using a finger on a touch sensitive surface or while performing a mid-air gesture, such as in a pointing posture or while holding an input device, etc. In examples, handwriting can include printing, cursive writing, pre-cursive writing, characters (e.g., Chinese characters) , numbers, symbols, punctuation marks etc.

In the present disclosure, a “stroke” can mean: a curved or straight line of various sizes or directions or having a defined a displacement or displacement trajectory. In examples, a series or sequence of strokes may be combined to form any number of patterns, shapes, letters, characters, symbols, numbers, drawings etc. In examples, “stroke information” can mean: positional information such as a set or sequence of coordinates, for example, corresponding to points of contact with a touch sensitive surface and optionally, describing a start position and an end position and a displacement or displacement trajectory of the stroke. In examples, stroke information may also include time information, for example, time stamps associated a start and end of a stroke, for example, for determining stroke order.

In the present disclosure, a “rich stroke” can mean: a stroke that is associated with additional features or characteristics for describing the stroke in addition to a standard positional information. In examples, “rich stroke information” may include positional information for a stroke as well as information related to the style of the stroke or related to how the stroke was generated to create a unique profile of the stroke, for example, including pressure, force (e.g., normal or shear) , acceleration, velocity or stroke thickness, with respect to time. In examples, rich stroke information may also include information about digital pen device orientation, for example, whether the digital pen is oriented normal to the touch sensitive surface or orientated at an angle to the touch sensitive surface (e.g., pen shear) .

In the present disclosure, a “small stroke” or a “small stroke input” can mean: A stroke associated with a very small displacement. Examples of small stroke inputs may include some punctuation (e.g., period, a comma, a colon, a semi-colon, an apostrophe, a quotation mark, a hyphen, ) or elements of punctuation (e.g. question mark, exclamation mark) or symbols (e.g., degree symbol, ellipsis etc. ) , among others. In examples, a small stroke input may be a single stroke input, for example, where the where a user does not remove the pen tip from the touchscreen surface to generate the input, for example, to generate a period, a comma, an apostrophe etc. In examples, a small stroke input may be a multi stroke input, for example, where a user removes the pen tip from the surface of the touch screen to perform multiple small strokes corresponding to the same character or symbol, such as for a colon, a semi-colon, a quotation mark, an ellipsis etc. In examples, multi strokes can be separated into multiple single strokes, for example, a semi-colon “; ” consists of a period “. ” and a comma “, ” .

In the present disclosure, a “pressure-varying stroke” or a “pressure-varied stroke” can mean, a stroke input where the magnitude of the pressure applied to the touch sensitive surface changes over time. In some examples, a pressure-varying stroke may be a “pressure-varying small stroke” , where the applied pressure changes over time but the stroke has a very small displacement. In some examples, a pressure-varying stroke may cause the stroke to also have a stroke thickness that changes over time, in response to the change in pressure. For example, an applied pressure may increase during the stroke, for example, causing the stroke thickness to change from thin to thick, or the applied pressure may decrease during the stroke, for example, causing the stroke thickness to change from thick to thin.

In the present disclosure, a “pressure profile” or a “pressure pattern” can mean: a feature of a stroke generation gesture that describes the magnitude of pressure applied to a touch sensitive surface of an electronic device over a specified time duration. For example, the pressure profile may capture an increase or decrease in pressure over time corresponding to one or more pressure peaks, among others.

In the present disclosure, a “positional profile” or a “positional pattern” can mean: a feature of a stroke generation gesture that describes the movement of the tip of a digital input device on a touch sensitive surface of an electronic device as a set or sequence of 2D coordinates, over a specified time duration.

In the present disclosure, a “pressure-assisted gesture” can mean: a gesture, such as a stroke generation gesture, performed by a user holding a digital pen device involving one or more of: a pressure profile input (e.g., where the pressure is applied normal to the touch sensitive surface at the point of contact of the digital pen device) , a positional profile input or a shear force profile input (e.g., where the force input is tangential to the touch sensitive surface with components in x and y) and which is associated with a specific UI interaction or control function of an electronic device.

In the present disclosure, a “stationary pressure-assisted gesture” can mean: a gesture that includes a pressure profile input at a static location (x, y) on the touch sensitive surface.

In the present disclosure, a “compound pressure-assisted gesture” can mean: a gesture that combines both a pressure profile input and a positional profile input.

In the present disclosure, “shear pressure-assisted gesture” can mean: a gesture that includes a pressure profile and a shear force profile input (e.g., including radial and axial force components in x and y) and optionally a positional profile input

Other terms used in the present disclosure may be introduced and defined in the following description.

FIG. 1 is a block diagram illustrating a simplified example implementation of a computing system 100 that is suitable for implementing embodiments described herein. Examples of the present disclosure may be implemented in other computing systems, which may include components different from those discussed below. The computing system 100 may be used to execute instructions for pressure-assisted gesture recognition, using any of the examples described herein.

The computing system 100 includes at least one processor 102, such as a central processing unit, a microprocessor, a digital signal processor, an application-specific integrated circuit (ASIC) , a field-programmable gate array (FPGA) , a dedicated logic circuitry, a dedicated artificial intelligence processor unit, a graphics processing unit (GPU) , a tensor processing unit (TPU) , a neural processing unit (NPU) , a hardware accelerator, or combinations thereof.

The computing system 100 may include an input/output (I/O) interface 104, which may enable interfacing with a digital pen device 140 and an electronic device 150. In the example shown, the digital pen device 140 may include input from a pressure/force sensor 142 and a motion sensor 144 (e.g., an accelerometer, a gyroscope, a tiltmeter etc. ) , among other sensors. In the example shown, pressure/force sensor 142 and a motion sensor 144 are shown as optional. In the example shown, the electronic device 150 may include a capacitive touch sensitive surface 152. In examples, the I/O interface 104, may enable interfacing with an input device 106 (e.g., a keyboard, a mouse, a camera, a microphone, and/or a keypad) and/or an output device 108 (e.g., a display, a speaker and/or a printer) . In the example shown, the input device 106 and output device 108 are shown as optional and external to the computing system 100.

The computing system 100 may include an optional communications interface 118 for wired or wireless communication with other computing systems (e.g., other computing systems in a network) . The communications interface 118 may include wired links (e.g., Ethernet cable) and/or wireless links (e.g., one or more antennas) for intra-network and/or inter-network communications.

The computing system 100 may include one or more memories 120 (collectively referred to as “memory 120” ) , which may include a volatile or non-volatile memory (e.g., a flash memory, a random access memory (RAM) , and/or a read-only memory (ROM) ) . The non-transitory memory 120 may store instructions 122 for execution by the processor 102, such as to carry out examples described in the present disclosure. For example, the memory 120 may store instructions for implementing any of the networks and methods disclosed herein. The memory 120 may include other software instructions, such as for implementing an operating system (OS) and other applications or functions. The instructions 122 can include instructions for implementing and operating the pressure-assisted gesture recognition system 200 described below with reference to FIG 2, instructions for implementing an optical character recognition (OCR) application 260, among other applications. The memory 120 may also store other data 124, information, rules, policies, and machine-executable instructions described herein, for example, including, rich stroke information 230.

In some examples, the computing system 100 may also include one or more electronic storage units (not shown) , such as a solid state drive, a hard disk drive, a magnetic disk drive and/or an optical disk drive. In some examples, data and/or instructions may be provided by an external memory (e.g., an external drive in wired or wireless communication with the computing system 100) or may be provided by a transitory or non-transitory computer-readable medium. Examples of non-transitory computer readable media include a RAM, a ROM, an erasable programmable ROM (EPROM) , an electrically erasable programmable ROM (EEPROM) , a flash memory, a CD-ROM, or other portable memory storage. The storage units and/or external memory may be used in conjunction with memory 120 to implement data storage, retrieval, and caching functions of the computing system 100. The components of the computing system 100 may communicate with each other via a bus, for example.

Although FIG. 1 shows a single instance of each component, there may be multiple instances of each component in the computing system 100. Further, although the computing system 100 is illustrated as a single block, the computing system 100 may be a single physical machine or device (e.g., implemented as a single computing device, such as a single workstation, single end user device, single server, etc. ) , and may include mobile communications devices (smartphones) , laptop computers, tablets, desktop computers, vehicle driver assistance systems, smart appliances, wearable devices, interactive kiosks, among others. In some embodiments, the computing system 100 may comprise a plurality of physical machines or devices (e.g., implemented as a cluster of machines, server, or devices) . In some embodiments, the computing system 100 may be a virtualized computing system (e.g., a virtual machine, a virtual server) emulated on a cluster of physical machines or by a cloud computing system.

FIG. 2 shows a block diagram of an example pressure-assisted gesture recognition system 200, in accordance with examples of the present disclosure. The pressure-assisted gesture recognition system 200 may be a software that is implemented in the computing system 100 of FIG. 1, in which the processor 102 is configured to execute instructions of the pressure-assisted gesture recognition system 200 stored in the memory 120. The pressure-assisted gesture recognition system 200 includes a processor 220 and a pressure-gesture recognizer 240.

The pressure-assisted gesture recognition system 200 receives an input of a stroke generation gesture 210 and outputs a pressure-assisted gesture 250. In examples, the pressure-assisted gesture 250 may allow a user to interact with the electronic device 150 by operating or controlling functions of an electronic device 150, for example, as described in example embodiments below. In examples, the stroke generation gesture 210 may be representative of a movement of a tip portion of a digital pen device 140 in contact with a touch sensitive surface of the electronic device 150. For example, the capacitive touch sensitive surface 152 may capture 2D positional information corresponding to points of contact of the digital pen device 140 on the capacitive touch sensitive surface 152 in response to the stroke generation 210. In examples, the stroke generation gesture 210 may include a handwriting gesture, a drawing gesture, a text editing gesture, a User Interface (UI) interaction gesture, a mode switching gesture or a user identification gesture, among others.

In some examples, the stroke generation gesture 210 is received by a processor 220 to generate rich stroke information 230. In examples, rich stroke information 230 may include positional information 232 as a set of coordinates (e.g., as a point cloud) mapped to the capacitive touch sensitive surface 152 that are associated with the stroke generation gesture 210, as well as information related to the style of the stroke or how the stroke was generated, for example, including pressure, force (e.g., normal or shear) , acceleration, velocity or stroke thickness information, and optionally, information about digital pen device 140 orientation (e.g., pen shear) as a function of time. In some embodiments, for example, the rich stroke information 230 may include pressure information 234 as a pressure profile, for example, describing a time-varying measure of a magnitude of pressure applied by the digital pen device 140 to the capacitive touch sensitive surface 152. In other embodiments, for example, the rich stroke information 230 may also include pseudo-shear information 236 describing information about tangential forces applied to the capacitive touch sensitive surface 152 by the digital pen device 140 during the stroke generation gesture 210 (described below with respect to FIG. 3D) . In other embodiments, for example, the rich stroke information 230 may also include motion information 238 describing information about position or orientation of the digital pen device 140 during the stroke generation gesture 210, for example, captured by a motion sensor 144.

FIG. 3A-3D illustrate an example embodiment of a digital pen device 140, in accordance with examples of the present disclosure. FIG. 3A illustrates a side view of an example embodiment of the digital pen device 140 including a flexible tip 300 in contact with the capacitive touch sensitive surface 152, producing a corresponding capacitive image 330a. In examples, the digital pen device 140 of FIG. 3A is oriented normal to the surface of the capacitive touch sensitive surface 152.

In some embodiments, for example, the digital pen device 140 may have a shaft 305 and a flexible tip 300 that is configured to sense shear forces applied to the capacitive touch sensitive surface 152. In examples, the flexible tip 300 may include a tip electrode 310 at the distal end of the flexible tip 300 for contacting the capacitive touch sensitive surface 152, and a ring electrode 320 surrounding the tip. In examples, the tip electrode 310 and ring electrode 320 may be any suitable conductive material, for example, metal, among others. In examples, a capacitive touch screen draws small electrical charges to a point of contact, and functions as a capacitor in the region of contact. In some examples, in response to the digital pen device 140 being placed in contact with the capacitive touch sensitive surface 152, a change in the capacitance and electrostatic field in the capacitive panel of the touch sensitive surface provides 2D location information corresponding to the contact position. In examples, when the digital pen device 140 is normal to the capacitive touch sensitive surface 152, the flexible tip electrode 310 and the ring electrode 320 may generate a capacitive image 330a on the capacitive touch sensitive surface 152, for example, with the location of the tip electrode 310 inducing a capacitive charge on the touch screen at the center 332 of the capacitive image 330a and the ring electrode 320 inducing a capacitive charge in the shape of an outer circle 334 in the capacitive image 330a. In some embodiments, for example, a pressure/force sensor 142 (e.g., one or more strain gauges attached to the tip electrode) may also be included at the distal end of the digital pen device 140, for directly measuring forces at the pen tip, for example, for measuring normal and shear forces. In some embodiments, for example, a motion sensor 144 (not shown) may also be housed within the shaft of the digital pen device 140 for detecting movement and/or orientation of the digital pen device 140. In examples, the shaft 305 may include control circuitry, for example, for transmitting to or receiving signals from computing system 100 of FIG. 1, or for processing signals received from any sensors of the digital pen device 110, among others.

FIG. 3B illustrates a side view of an example embodiment of the digital pen device 140 and a corresponding capacitive image 330b, in accordance with examples of the present disclosure. In examples, the digital pen device of FIG. 3B is in contact with the capacitive touch sensitive surface 152 and oriented at an angle 325 to the capacitive touch sensitive surface 152. In examples, when the digital pen device 140 is at an angle 325 with respect to the capacitive touch sensitive surface 152, the tip electrode 310 and the ring electrode 320 may generate a capacitive image 330b on the capacitive touch sensitive surface 152, for example, with the tip electrode 310 inducing a capacitive charge on the touch screen at a location 336 and the ring electrode 320 inducing a capacitive charge in the shape of an ellipse 338 in the capacitive image 330b. In examples, a relationship between features of the capacitive image 330b and the tilt angle of the digital pen device 140 can be developed, for example, based on the bending equation for a beam and known material properties of the flexible tip 300 (e.g., elastic modulus, among others) . In examples, using this relationship, by measuring the distance between the center of the tip electrode 336 and the center of the ring electrode ellipse 338 in the capacitive image 330b as well as the length of the ring electrode ellipse 338, it is possible to determine the tilt angle of the digital pen device 140 and any corresponding bending of the flexible tip 300 which may be applying shear forces τx and τy on the surface of the touch sensitive surface 152.

FIG. 3C illustrates a magnified view of an example embodiment of the tip electrode 310 of the digital pen device 140, in accordance with examples of the present disclosure. In examples, the tip electrode 310 may protrude from the distal end of the digital pen device 140 and may be composed of a flexible material that is configured to bend in response to contact with the capacitive touch sensitive surface 152, for example, at any angle other than normal to the surface of the capacitive touch sensitive surface 152.

FIG. 3D illustrates an example approach for computing a pseudo shear vector for a digital pen device 140 in contact with a capacitive touch sensitive surface 152 of an electrical device 150, in accordance with examples of the present disclosure. In examples, computing the pseudo-shear vector leverages the lack of friction between the touch sensitive screen and the digital pen device, therefore, a small pen shear will cause movement in the same direction. In examples, a direction vector may first be obtained for the digital pen device 140. In examples, coordinates for two consecutive stroke points of a stroke generation gesture on the capacitive touch sensitive surface 152 may be determined to generate an intermediate vector. In examples, the intermediate vector may then be normalized to generate the direction vector. In examples, the normalized direction vector is then multiplied by the amplitude of the pressure measurement, where the amplitude of the pressure measurement is an average of the pressure measured at the two consecutive stroke points of the stroke generation gesture, to generate a directional tangential force vector (e.g., pseudo shear τx 340 and τy 350) .

Returning to FIG 2, in some embodiments, for example, the rich stroke information 230 may include pressure information 234 determined by one or more force sensors arranged in a 2D array within the touch sensitive surface, for example, as a pressure pad. In examples, the pressure pad may measure a pressure pattern corresponding to the contact of the digital pen device 140 on the touch sensitive surface in response to the stroke generation gesture 210. In some examples, the pressure information 234 may be represented by a sequence of force measurements, where the value of the force measurements may be proportional to the magnitude of the applied force by the digital pen device 140 at each point in the pressure array during the stroke generation gesture 210. In this regard, the stroke information may be considered to be three-dimensional (3D) , including both 2D positional information 232 and pressure information 234. In some examples, a benefit of generating positional information 232 and pressure information 234 with a capacitive touch sensitive surface 152 is that touch sensitive surfaces require lower power and are readily embedded into many surfaces on commercial devices, for example, tablets, laptops, smartphones or dual-screen devices, compared to a pressure pad that may require greater power requirements.

In examples, the rich stroke information 230 may also include motion information 238. In some embodiments, for example, the digital pen device 140 may include a motion sensor 144, for example, an accelerometer or an inertial measurement unit (IMU) to detect the movement of the digital pen device 140 in response to a stroke generation gesture 210. In some examples, the motion sensor 144 may be a 3 degree-of-freedom (3DoF) IMU, a 6 degree-of-freedom (6DoF) IMU or a 9 degree-of-freedom (9DoF) IMU, where the IMU may comprise an accelerometer that measures translational acceleration in 3-axes, a gyroscope that measures rotational velocity or acceleration in another 3-axes or optionally a magnetometer that measures a magnetic field strength in 3-axes. In some examples, the motion information 238 generated by the motion sensing device 220 during a hand action 210 may be represented by 3 channels of time-series translational acceleration measurements (e.g. force or acceleration) , 3 channels of time-series rotational velocity measurements (e.g. angular rate) and optionally 3 channels of time-series magnetic field measurements (e.g. orientation) , corresponding to movement of the digital pen device 140 in response to the stroke generation gesture 210.

In examples, the rich stroke information 230 may be input to a pressure-gesture recognizer 240 to generate a pressure-assisted gesture 250. In examples, a pressure-assisted gesture 250 may be a “stationary pressure-assisted gesture” , comprising a pressure profile generated at a single point of contact on the touch sensitive surface. In examples, a pressure- assisted gesture 250 may be a “compound pressure-assisted gesture” , where at least one pressure-varying stroke is associated with a pre-defined pressure pattern and a pre-defined positional pattern. In examples, a pressure-assisted gesture 250 may be a “shear pressure-assisted gesture” , where a pressure-varying stroke is associated with a pre-defined pressure pattern that includes tangential force components and may also be associated with a pre-defined positional pattern. In some embodiments, the pressure-gesture recognizer 240 may use different methods for generating a pressure-assisted gesture 250 depending on the application.

In one embodiment, the pressure-gesture recognizer 240 may pre-process the rich stroke information 230 for a candidate stroke point cloud Ci associated with a stroke generation gesture 210, by resampling, rotating and scaling the positional information 232. In examples, the pressure information 234 may be input as a new dimension to a traditional stroke recognizer (for example, a stroke recognizer using a Euclidian distance method for comparing candidate stroke point cloud Ci with stored template images Ti) to determine the average distance di between corresponding points using equation 1:

where C is a point cloud for the candidate stroke, k is an index of the stroke point in the point cloud and N is the number of points in the point cloud. In another embodiment, the pressure-gesture recognizer 240 may pre-process the rich stroke information 230 for a candidate stroke Ci associated with a stroke generation gesture 210, using a time series analysis, such as dynamic time warping (DTW) , to measure similarity between two temporal sequences. For example, DTW may be run on the pressure information 234 over time, to determine a weighted DTW distance W‖D_DTW‖, which can then be added to the Euclidean distance determined using a traditional stroke recognizer, for each point in the candidate stroke Ci using equation 2:
d_i=‖D_x, y, p‖=‖D_x, y‖+W‖D_DTW‖ (2)

In another embodiment, the pressure-gesture recognizer 240 may pre-process the rich stroke information 230 for a candidate stroke C_i associated with a stroke generation gesture 210, including combining the pseudo-shear information 236 with the direction of movement to create directional forces τ_x and τ_y in X and Y directions, respectively. In examples, the directional forces τ_x and τ_y can be input to a traditional stroke recognizer for computing the Euclidean distance for the candidate stroke C_i compared to stored template images T_i to determine the average distance d_i between corresponding points using equation 3:
d_i=‖D_τx, τy‖ (3)

In another embodiment, the pressure-gesture recognizer 240 may be a trained machine learning (ML) model that is trained to recognize pressure-assisted gestures for a given input stroke image. In examples, the ML model may be a neural network model. In examples, the input layer for the ML model is fed a user drawing on the canvas, where it is downscaled to a M x M grid and feed into a neural network with N layers. IN examples, the neural network may be a sequential convolutional neural network, or another model may be used. In examples, the sequential constructor may consist of three types of layers, including convolutional, max pooling and softmax.

In some embodiments, for example, the pressure-gesture recognizer 240 may be configured to interface with an optical character recognition (OCR) application 260 of the computing system 100 of FIG. 1. In examples, incorporating rich stroke information 230 via the pressure-gesture recognizer 240 into OCR workflows may improve the accuracy of recognized characters 270 detected by the OCR application 260, as described with respect to FIG. 4-8 below.

FIG. 4 is a flowchart illustrating an example algorithm 400 for stroke recognition, including small strokes, using pressure-assisted gestures 250, in accordance with an example embodiment of the present disclosure. In the example embodiment, stroke generation gestures 210 may be used to generate new text, or to edit an existing text, for example, small stroke inputs may be added in the form of punctuation to a text transcript generated by an automatic speech recognition (ASR) application. In examples, the small stroke recognition algorithm 400 is compatible with existing optical character recognition (OCR) functionality.

In examples, at step 405, the stroke generation gesture 210 is captured, for example, using a stroke capture timeout period, and corresponding rich stroke information 230 is generated. If the detected stroke generation gesture 210 captured within the timeout period includes only one stroke, the algorithm proceeds to step 410. If the detected stroke generation gesture 210 captured within the timeout period includes more than one stroke, the algorithm proceeds to step 415.

At step 410, rich stroke information 230 for the stroke generation gesture 210 is evaluated, for example, the stroke is analyzed with respect to position pattern, pressure pattern, stroke position relative to neighboring text, stroke size (e.g., Δx and Δy) . In examples, the stroke size may be compared to a threshold value to determine whether the stroke is a small stroke and a corresponding pressure-assisted gesture 250 may be generated. In examples, if the stroke is classified as a small stroke, the OCR application may proceed to feed a corresponding stationary pressure-assisted gesture into step 420. In examples, if the comparison with the threshold value indicates that the stroke is not a small stroke, a compound pressure-assisted gesture or a shear pressure-assisted gesture may be fed into step 420.

At step 415, the OCR application may determine how many strokes are associated with the stroke generation gesture 210, for example, a stroke generation gesture 210 for the letter “T” may include two strokes whereas a stroke generation gesture 210 for the letter “N” may include three strokes. The multi-stroke information may then be fed into step 425 where the first stroke may be identified and either fed into step 410 to generate rich stroke information 230 for the stroke, or fed into step 435 to determine whether the stroke is a component of a multi-stroke punctuation input before proceeding to step 410.

At step 420, the OCR may determine, with the help of the provided pressure-assisted gesture 250, whether the stroke is a punctuation input or a character input, and if it is a punctuation input, which punctuation mark the input is associated with. In examples, a punctuation input may indicate that the stroke is a single stroke punctuation mark (e.g., period, comma etc. ) or a component of a multi-stroke punctuation mark (e.g., colon, semi-colon etc. ) whereas a character input may represent any complete character (e.g., letters, numbers, symbols etc. ) including complete punctuation marks. If the stroke is determined to be a punctuation input, the algorithm proceeds to step 435 to determine whether the punctuation input is associated with a multiple-stroke punctuation mark. At step 435, if the stroke is determined to be a single stroke of a multiple-stroke punctuation mark, the algorithm returns with this information to step 410. Alternatively, at step 435, if the stroke is determined to be a complete multiple-stroke punctuation mark, the algorithm proceeds to step 430. In examples, if the stroke is determined to be a character input, the algorithm proceeds to step 430.

At step 430, the OCR application may parse the text for punctuation, since punctuation can occur at the beginning, middle or end of a sentence. In examples, OCR identifies a series of characters followed by a potential punctuation and its location with respect to the characters; however, the OCR cannot accurately determine the exact punctuation, therefore the algorithm proceeds to step 445 for detailed analysis on potential punctuation buried in the text, for example, to determine the appropriate punctuation.

Alternatively, at step 430, if only a character is detected, the character may be inputted into the text at step 440, for example, excluding any punctuation.

At step 445, potential punctuation identified by the OCR may be returned to step 425 for detailed analysis. In examples, when a single or multi-stroke punctuation mark is detected, the punctuation may be identified at step 445 based on a stroke identification (ID) and, for example, passed on to step 450 for insertion into the text. In examples, the algorithm may serve to assist the OCR to identify the exact punctuation for insertion at step 450, for example, with respect to the OCR recognized characters, in the middle of a sentence.

In examples, by incorporating pressure-assisted gestures into the OCR workflow, punctuation and other small strokes can be more accurately identified. For example, when the OCR comes across a stroke that potentially represents a punctuation input, the OCR can incorporate additional rich stroke information 230 to the character recognition workflow to ensure that the character is correctly recognized.

In another example embodiment of the present disclosure, the pressure-gesture recognizer 240 may be configured to interface with an optical character recognition (OCR) application 260 of the computing system 100 of FIG. 1 for improving character segmentation, as described with respect to FIGs 5A-C and FIG. 6 below. Example handwriting samples of FIGs 5A-C are provided in Privitera, C.M., & Plamondon, R., (1995, August) , A system for scanning and segmenting cursively handwritten words into basic strokes, In Proceedings of 3rd International Conference on Document Analysis and Recognition (Vol. 2, pp. 1047-1050) , IEEE, the entirety of which is hereby incorporated by reference. FIG. 5A illustrates an example of handwriting where individual letters are joined, making it difficult for traditional OCR applications to segment words into component characters. Segmentation means the separation of clear character area from non-character area in the given image. Traditional segmentation approaches include thresholding of the image, skeletonization of the image and finally, pruning. Skeletonization is performed to reduce the character area to just one-pixel line, however, during this process, some unwanted branches may be added to the output skeletons. Pruning refers to the removal of these erroneous branches or spurs from the skeletons. In examples, joined-up writing is difficult to recognize and easily misinterpreted by OCR in all languages.

In examples, the shape of a handwritten character depends on its neighbors. In letters with multiple stroke intersections (e.g., “t” and “x” ) and/or combinations of strokes and dots (e.g., “i” and “j” ) the order of pen strokes is ambiguous. In examples, rich stroke information 230 including pressure information 234 obtained for a stroke generation gesture 210 can be used to segment different cursive or joined characters, where variation in pressure, for example, pressure patterns such as “high-low-high” may help to inform the segmentation. Similarly, pressure information rich stroke information 230 may also be incorporated into curvature analysis for character segmentation, for example, by identifying points of first contact 505 and last contact 510 between the digital pen device 140 and the touch sensitive surface, as shown in FIG. 5B, or identifying curvature points 515 within a stroke where a maximum curvature is detected, as shown in FIG. 5C.

FIG. 6 is a flowchart illustrating an example algorithm 600 for character segmentation using pressure-assisted gestures 250, in accordance with an example embodiment of the present disclosure. In the example embodiment, pressure-assisted gestures 250 may be used to segment characters in handwritten text that may not have clear separation between each letter, for example, in cursive handwriting or handwriting with joined characters, for example, as shown in the example text depicted in FIG. 5A. In examples, the character segmentation algorithm 600 is compatible with existing optical character recognition (OCR) functionality.

In examples, at step 610, the stroke generation gesture 210 is captured, for example, using a stroke capture timeout period. At step 620, rich stroke information 230 corresponding to the captured stroke generation gesture 210 is generated.

At step 630, curvature points 515 may be recognized based on the rich stroke information 230 for the stroke generation gesture 210. In examples, curvature points 515 may be detected using a curvature function of the word contour and rich stroke information 230 such as pressure information 234. In examples, points of first contact 505 and last contact 510 between the digital pen device 140 and the touch sensitive surface in the stroke generation gesture 210 may also be determined, for example, using pressure information 234, and labeled.

At step 640, a subset of the curvature points 515 may be selected for input to the segmentation process of the OCR application 260. At step 650, the selected curvature points 515 may be used by the OCR application 260 to identify potential characters. Finally, at step 660, the OCR application may finalize the character segmentation by recognizing all characters.

FIG. 7 is a flowchart illustrating an example algorithm 700 for recognizing connections between adjacent characters in handwritten text using pressure-assisted gestures 250, in accordance with an example embodiment of the present disclosure. In the example embodiment, pressure-assisted gestures 250 may be used to determine connections in handwritten text that may not have clear separation between each letter, for example, in cursive handwriting or handwriting with joined characters, for improved OCR. In examples, the character segmentation algorithm 700 is compatible with existing optical character recognition (OCR) functionality.

In examples, at step 705, a user may complete an onboarding process to gather examples of stroke generation gestures 210 and corresponding rich stroke information 230 for the user, to train a machine learning model.

At step 710, a stroke generation gesture 210 is captured, for example, using a stroke capture timeout period, and corresponding rich stroke information 230 is generated for individual strokes in the stroke generation gesture 210.

At step 715, rich stroke information 230 for the stroke generation gesture 210 is evaluated, for example, with respect to position pattern, pressure pattern, stroke position relative to neighboring text, stroke size (e.g., Δx and Δy) . In examples, the OCR application 260 may receive the rich stroke information 230 for the stroke generation gesture 210 and determine whether the handwritten text includes joined-up strokes 715a or connection strokes 715b. If the handwritten text is identified as including joined-up strokes 715a, the algorithm proceeds to step 720 where the OCR application identifies the individual strokes within the joined-up strokes based on the rich stroke information 230. For example, individual strokes may be detected based on changes in a pressure pattern from low pressure to high pressure, or, alternatively, from high pressure to low pressure, among others. In examples, once an individual stroke is detected, the algorithm proceeds to step 730 where information obtained from the rich stroke information 230 related to stroke order is sent to the OCR character library for processing. If the handwritten text is identified as including connection strokes 715b, the algorithm proceeds to step 725 where the OCR application separates the characters based on the rich stroke information 230. For example, a connection strokes may be detected based on changes in a pressure pattern from low pressure to high pressure, or, alternatively, from high pressure to low pressure, among others.

In examples, steps 735 to 750 may represent steps performed by the OCR application 260 to recognize strokes within a stroke generation gesture 210 and to recognize characters based on the recognized strokes, for example, as described with respect to FIG. 4.

At step 755 the OCR application 260 determines the best match for the recognized character and may populate the text with the recognized character. In examples, at step 760, the OCR application 260 may present alternative characters to a user.

FIG. 8 is a flowchart illustrating an example algorithm 800 for recommending potential characters in handwritten text in real-time using pressure-assisted gestures 250 and stroke order, in accordance with an example embodiment of the present disclosure. In the example embodiment, pressure-assisted gestures 250 and stroke order may be used to predict a most likely character as it is being written, and enable an OCR application 260 to auto-complete the character. In examples, the character segmentation algorithm 800 is compatible with existing optical character recognition (OCR) functionality.

In examples, at step 805, a user may complete an onboarding process to gather examples of stroke generation gestures 210 and corresponding rich stroke information 230 for the user, to train a machine learning model.

At step 810, a stroke generation gesture 210 is captured, for example, using a stroke capture timeout period, and corresponding rich stroke information 230 is generated for individual strokes in the stroke generation gesture 210.

At step 815, rich stroke information 230 for the stroke generation gesture 210 is evaluated, for example, with respect to position pattern, pressure pattern, stroke position relative to neighboring text, stroke size (e.g., Δx and Δy) . In examples, At step 810, the OCR application 260 may receive the rich stroke information 230 for the stroke generation gesture 210 within a pre-defined bounding box, and evaluate, for example, whether the character (e.g., a Chinese character) is complete. In examples, the OCR application 260 may determine whether the character is complete based on current stroke order, and stroke occupancy within the pre-defined bounding box for the character. If the character is determined to be in-progress, the algorithm proceeds to step 815. If the character is determined to be complete, the algorithm proceeds to complete a sequence of steps 745, 750, 755 and 760 as described with respect to FIG. 7.

In step 820, the stroke is identified based on the stroke order and the rich stroke information 230, for example, using pressure information 234 and/or pseudo-shear information 236.

In step 825, information obtained from the rich stroke information 230 related to stroke order is sent to the OCR character library for processing.

At step 830, the OCR application 260 may predict potential character results using the trained machine learning model, based on the current in-progress strokes and based on the rich stroke information 230.

At step 835, the OCR application 260 may provide one or more recommended characters based on the predicted characters. In addition, if the stroke order is enough to identify the character, the OCR application 260 may auto-complete the character at step 840.

In addition to user interface interactions related to handwriting, for example, using the digital pen device 140 in contact with a touch sensitive surface, many other functionalities of a digital pen device exist for user interface interactions. In examples, configuring digital pen functionality is commonly performed using menus, for example, for the purpose of mode switching or to enable state changes, for example, holding a current state or enabling temporary states. In examples, pressure-assisted gestures 250 may present opportunities for user interface interactions including menu selections, mode switching, state changes, among others.

FIG. 9 is a schematic diagram illustrating an example algorithm 900 for state transition between drag and shear, using pressure-assisted gestures 250, in accordance with an example embodiment of the present disclosure. In the example embodiment, pressure-assisted gestures 250 may be used to distinguish between a shear force operation and a drag gesture, for improved interaction with graphical user interfaces (GUI) .

In examples, at step 910, a digital pen device 140 comes into contact with a touch sensitive surface. In examples, a calibration may be performed at the onset of contact with the touch sensitive surface and the pressure-assisted gesture recognition system 200 may be considered to be operating within a “contact state” .

At step 920, a change in pressure associated with the digital pen 140 in contact with the touch sensitive surface may be measured, for example, a user may “press” or “unpress” the tip of the digital pen device 140 to increase or decrease the force applied to the touch sensitive surface, while still maintaining contact with the touch sensitive surface. In examples, when the force applied by the digital pen device 140 on the touch sensitive surface exceeds a predefined threshold, the pressure-assisted gesture recognition system 200 may transition to operating within a “pressed state” .

In examples, a speed of movement associated with the digital pen 140 in contact with the touch sensitive surface may be measured, for example, a user may quickly move the pen (e.g., “drag” ) or the user may slowly move the digital pen device 100 (e.g., “shear” ) , while still maintaining contact with the touch sensitive surface. In examples, the speed of movement of the digital pen device 140 may be compared to a threshold value. At step 930, when a speed of movement of the digital pen device 140 is greater than the threshold value, the pressure-assisted gesture recognition system 200 may transition to operating within a “drag state” . At step 940, when a speed of movement of the digital pen device 140 is less than the threshold value, the pressure-assisted gesture recognition system 200 may transition to operating within a “shear state” .

In examples, in each of the operating states, specific user interactions may be performed and functionalities may be enabled. In examples, the pressure-assisted gesture recognition system 200 may exit either the “drag state” or the “shear state” and return to the “contact state” by reducing the force applied to the touch sensitive surface by the digital pen device 140, for example, by “unpressing” the tip of the digital pen device 140.

In examples, user interface interactions while the pressure-assisted gesture recognition system 200 is operating in either a contact state, pressed state, drag state or shear state may serve a number of applications. A list of non-limiting applications includes, for example, panning or zooming, user interface object selection, document or file management, cropping, drawing, painting, character or text editing, or handwriting, among others. In one embodiment, for example, a hard drag across a document may instruct “Tearing off” a copy of the document, or a copy of a portion of a document, or extracting a portion of the document, among others. Examples of these and other applications are described with respect to FIG. 10-12 below.

FIG. 10 is a flowchart illustrating an example algorithm 1000 for UI interaction using pressure-assisted gestures 250, for example, involving UI object selection or text editing, in accordance with an example embodiment of the present disclosure. In examples, when editing text on a touch sensitive device, a user may use circle or strikethrough gestures to select a word or a sentence; however, if the stroke gesture extends beyond the desired character or word, additional text may be unintentionally selected. Alternatively, if a strikethrough line selection accidentally extends beyond a last selected word, the next word in the text may be unintentionally selected. In the example embodiment, the pressure-assisted gesture recognition system 200 may continuously monitor movement of the digital pen device 140 while it is in contact with the capacitive touch sensitive surface 152 in order to detect specific pressure-assisted gestures 250 configured to control respective functions of the user interface. In examples, incorporating rich stroke information 230 into user interface gesture controls may help optimize UI interaction.

In examples, at step 1005, the pressure-assisted gesture recognition system 200 may enter a pressure-assisted gesture input mode, for example, where the pressure-assisted gesture recognition system 200 may continuously monitor a user’s rich stroke input for pre-determined pressure-assisted gestures 250. In examples, the pressure-assisted gesture recognition system 200 may enter the pressure assisted gesture input mode in response to receiving an activation signal, for example, through a mode-switching operation, or using an activation gesture or menu selection, among others.

In examples, at step 1010, in response to monitoring the user’s rich stroke input, a pressure-assisted gesture 250 may be detected. In examples, pressure-assisted gestures 250 may be categorized based on any aspect of rich stroke information associated with the performed gesture, for example, stroke position, shape, pressure, shear, speed, acceleration etc., or digital pen device 140 orientation, or any combination thereof.

In examples, if the detected pressure-assisted gesture 250 has a smooth motion feature, for example, a circling gesture or a strikethrough gesture of a text editing operation, exhibiting continuous motion throughout the gesture, the algorithm will proceed to step 1015. In examples, if the detected pressure-assisted gesture 250 has a hesitation feature, the algorithm will proceed to step 1045. In examples, a hesitation feature may represent a pause at the end of a first stroke of a multi-stroke pressure-assisted gesture 250, where the pause may cause the pressure-assisted gesture recognition system 200 to provide a hint for completing the multi-stroke pressure-assisted gesture 250.

In examples, text editing operations may refer to UI interactions where a user can insert, change or delete text at the character level or including complete words and sentences. Unlike touch screen interactions using a finger contact, a digital pen device 140 can target a single character and instruct necessary edits. However, selecting a single character may be challenging, for example, depending on the resolution of the touch sensitive surface and/or the digital pen device 140, incorrect selection of characters or words may occur. Incorporating rich stroke information 230, for example, pseudo-shear information 136, into text editing gestures may improve selection accuracy.

In examples, at step 1015, the pressure-assisted gesture 250 having a smooth motion feature may be detected, for example, where completion of the pressure-assisted gesture 250 is signaled by a pen lift action (e.g., the digital pen device 140 is removed from contact with the touch sensitive surface) .

In examples, at step 1020, the pressure-assisted gesture 250 may be classified based on a set of pre-determined pressure-assisted gestures. For example, the pressure- assisted gesture 250 may be classified as a circling gesture or a strikethrough gesture performed as part of a text editing operation, among others.

In examples, at step 1025, the pressure-assisted gesture recognition system 200 may analyze underlying text content in a text editing operation to determine the intent of the text, along with rich stroke information 230 associated with the pressure-assisted gesture 250, based on the gesture classification. In examples, a degree of pen shear (e.g., as determined from the digital pen device 140 orientation) may inform whether a single character or an entire word is selected in response to a circling gesture. In other examples, a pressure profile or acceleration associated with a strikethrough gesture may inform the start and end of a text selection.

In examples, at step 1030, the pressure-assisted gesture recognition system 200 may interpret the rich stroke information 230 associated with the pressure-assisted gesture 250 to define a text selection, for example, to define the characters or strings that the user intended to select (e.g., by circling the text or striking the text, among others) .

In examples, at step 1035, a user interface of the electronic device 150 may display a UI menu, or another UI object or interface option for controlling a function of the electronic device, based on the defined selection. For example, in the context of a text editing operation, a UI menu or other UI object may be displayed enabling the user to select an operation to be performed on the selected portion of text, for example, to copy, cut or delete the selected portion of text, or to alter a style of the selected portion of text, among others.

In examples, at step 1040 the UI menu option may be selected, enabling the text editing operation to be performed. In examples, in response to the menu selection, the algorithm may proceed to step 1060 where the text content may be updated based on the selection.

In examples, at step 1045, a pressure-assisted gesture 250 having a hesitation motion feature may be detected, for example, where a pause or appreciable slowdown in gesture motion (e.g., based on a deceleration threshold) is detected after a first stroke of a multi-stroke gesture. At step 1050, in response to the hesitation motion feature, a visual hint of a potential next stroke gesture of the multi-stroke pressure-assisted stroke gesture 250 may be provided to a display of the electronic device 150. For example, a degree of pen shear may inform a gesture prediction, for example, by provide a visual hint of a predicted circle selection close point, or automatically closing a circle selection when the current stroke approaches a predicted end point, among others.

In examples, at step 1055, an input gesture may be performed, for example, by accepting the visual hint provided to the display or by completing another next stroke gesture of the multi-stroke pressure-assisted stroke gesture 250. In examples, in response to the performing the input gesture and completing the multi-stroke pressure-assisted stroke gesture 250, the algorithm may proceed to step 1060 where the text content may be updated based on the selection.

In other examples, rich stroke information 230 may support digital drawing or painting applications. In examples, it is difficult to add constraints to a freehand drawing while a user is generating new strokes. Using pressure-assisted gestures 250, for example, incorporating pseudo-shear information 236, a user may add and manipulate control points to drawing strokes as they are being generated. In other examples, a pressure-assisted gesture 250 may be used to convert a freehand stroke into a line segments, for example, using a hard drag gesture, among others.

FIG. s11A-F illustrate example embodiments of stationary pressure-assisted gestures for interacting with a user interface of an electronic device 150, in accordance with examples of the present disclosure. In examples, interaction designers often use different types of tapping or press inputs (e.g., as typical touch gestures) for interacting with touch sensitive devices. Example touch inputs may include a long press, a single-tap, a double-tap, a triple-tap, two finger press, etc. In examples, incorporating rich stroke information 230 into tap-based or otherwise stationary gesture controls may help to optimize UI interaction.

FIG. 11A provides an example of a stationary pressure-assisted gesture 1102 (e.g., a discrete tap gesture) and an associated pulse pressure profile 1100, for example, where a digital pen device 140 is brought into contact with a capacitive touch sensitive surface 152 and a corresponding pressure profile 1100 is obtained before the digital pen device 140 is removed from contact. In examples, the pressure profile 1100 resembles a rectangular pulse as pressure is measured at a constant magnitude for the duration of contact. Alternatively, by incorporating pressure-varying gestures, for example, varying pressure in a sequence from low-to-high-to-low, among other sequences, may add additional dimension to gesture controls.

FIG. 11B provides an example of a stationary pressure-assisted gesture 1104 and an associated pulse pressure profile 1100, for example, where a digital pen device 140 is brought into contact with a capacitive touch sensitive surface 152 and a corresponding pressure profile 1100 is obtained before the digital pen device 140 is removed from contact. In examples, the pressure profile 1100 resembles a Gaussian pulse representative of a controlled increase in pressure followed by a controlled decrease in pressure (e.g., a pen pressure tap) . In examples, stationary pressure-assisted gestures such as tap gestures may be used alone, or in combination with other pressure-assisted gestures 250 (e.g., as a compound pressure-assisted gesture) to control various functions of an electronic device 150, for example, in mode-switching operations or menu selection operations, among others.

FIG. 11C provides an example of a stationary pressure-assisted gesture 1106 and an associated pulse pressure profile 1100, for example, including three controlled pressure pulses. In examples, the stationary pressure-assisted gesture involving three pressure pulses may invoke a handwriting mode 1110 in the user interface.

FIG. 11D provides an example of a stationary pressure-assisted gesture 1108 and an associated pulse pressure profile 1100, for example, including two controlled pressure pulses. In examples, the stationary pressure-assisted gesture involving two pressure pulses may invoke a context menu 1120 in the user interface.

FIG. 11E provides an example of a compound pressure-assisted gesture 1130 for invoking a menu selection for a portion of text. In examples, the compound pressure-assisted gesture 1130 includes a first gesture 1132 involving a strikethrough gesture along a portion of text, a second gesture 1134 involving a stationary pressure-assisted gesture (e.g. a pen pressure tap) to invoke a menu, and a third gesture 1136 involving a menu selection.

FIG. 11F provides an example of a compound pressure-assisted gesture 1140 for performing a text editing operation. In examples, the compound pressure-assisted gesture 1140 includes a first gesture 1142 involving a squiggle gesture along a portion of text to instruct a deleting of the text, a second gesture 1144 involving a stationary pressure-assisted gesture (e.g. a pen pressure tap) to cancel or “undo” the delete instruction, and a third gesture 1146 involving a pen lift to complete the compound pressure-assisted gesture 1140.

FIG. 12 is a flowchart illustrating an example algorithm 1200 for enhanced personalization of stroke gestures using pressure-assisted gestures 250, in accordance with an example embodiment of the present disclosure. In examples, electronic devices that do not support fingerprint login currently use passwords or screen patterns for lock and unlock functions, which are difficult to remember and time-consuming to enter. In examples, multi-user devices may further complicate the process, requiring the user to switch accounts first before they can log in. In examples, incorporating rich stroke information 230 into gestures related to encryption/decryption or login operations can improve efficiency and user experience.

In examples, at step 1210, a user may complete an onboarding process to gather examples of stroke generation gestures 210 and corresponding rich stroke information 230 for the user, associated with a personalized login operation, to train a machine learning model to detect a personalized login gesture. In examples, a personalized login gesture may include a signature or a handwriting sample, or a personalized drawing pattern or symbol, among others.

In examples, at step 1220, the pressure-assisted gesture recognition system 200 may enter a pressure-assisted gesture input mode, for example, where the pressure-assisted gesture recognition system 200 may continuously monitor a user’s rich stroke input, for example, in a profile analyzing region of the user interface (e.g., a login input box) for pre-determined pressure-assisted gestures 250. In examples, the pressure-assisted gesture recognition system 200 may enter the pressure assisted gesture input mode in response to receiving an activation signal, for example, through a mode-switching operation, or using an activation gesture or menu selection, among others.

In examples, at step 1230, the pressure-assisted gesture recognition system 200 may continuously monitor the user’s rich stroke input until the pen stroke input is completed, for example, as indicated by a pen lift gesture or a stroke capture timeout period, among others. In examples, a complete pen stroke input may be considered to be a pressure-assisted gesture 250.

In examples, at step 1240, in response to receiving the user’s pen stroke input, the rich stroke information 230 associated with the user’s pen stroke input may be analyzed, for example, with respect to position pattern, pressure pattern, pen shear, device motion etc., to determine whether the pressure-assisted gesture 250 matches a pre-defined personalized login gesture for the user. In examples, if the pressure-assisted gesture 250 matches the pre-defined personalized login gesture for the user, at step 1250, a user may be granted access to the device or user account and may proceed to a personalized user experience based on the application.

In other embodiments, for example, rich stroke information may be incorporated into an encryption operation, for example, to protect personalized or confidential documents or files. In examples, securing or encrypting note applications or documents currently requires password protection, which can be time consuming to enter via keyboard input, or may require mode-switching operations to enter a password. In examples, rich stroke information 230 may be collected for a user during normal use of an application, for example, while writing a personalized note in a note application, to identify a user through their handwriting. In examples, upon completion of the handwritten note, the pressure-assisted gesture recognition system 200 may prompt the user with an option for storing the note in a Personal Protected Mode, for example, by encrypting the file, or the pressure-assisted gesture recognition system 200 may automatically store the file in a personal protected folder.

FIG. 13 is a flowchart illustrating an example computer implemented method 1300 for interacting with an electronic device 150 having a touch sensitive surface configured to detect a stroke generation gesture 210 on the touch sensitive surface, in accordance with examples of the present disclosure. The method 1300 may be performed by the computing system 100. For example, the processor 102 may execute computer readable instructions (which may be stored in the memory 120) to cause the computing system 100 to perform the method 1300. The method 1300 may be performed using a single physical machine (e.g., a workstation or server) , a plurality of physical machines working together (e.g., a server cluster) , or cloud-based resources (e.g., using virtual resources on a cloud computing platform) .

Method 1300 begins with step 1302 in which a pressure-assisted gesture recognition system 200 receives a rich stroke information 230 corresponding to the stroke generation gesture 210. In examples, the stroke generation gesture 210 may be representative of a movement of a tip portion of a digital pen device 140 in contact with a capacitive touch sensitive surface 152 of the electronic device 150. In examples, the stroke generation gesture 210 may include a handwriting gesture, a drawing gesture, a text editing gesture, a User Interface (UI) interaction gesture, a mode switching gesture or a user identification gesture, among others

In examples, the rich stroke information 230 may include a 2D positional information 232 generated by the touch sensitive surface in response to detecting the stroke generation gesture, for example, as a set of coordinates corresponding to points of contact of the digital pen device 140 on the capacitive touch sensitive surface 152. In examples, the rich stroke information 230 may also include a pressure information 234 as a pressure profile, for example, describing a time-varying measure of a magnitude of pressure applied by the digital pen device 140 to the capacitive touch sensitive surface 152. In other embodiments, for example, the rich stroke information 230 may also include pseudo-shear information 236 describing information about tangential forces applied to the capacitive touch sensitive surface 152 by the digital pen device 140 and motion information 238 describing the position or orientation of the digital pen device 140 during the stroke generation gesture 210.

At step 1304, a pressure-gesture recognizer 240 may generate a pressure-assisted gesture 250 using the positional information 232 and the pressure information 234. Optionally, the pressure-gesture recognizer 240 may generate the pressure-assisted gesture 250 further based on the pseudo-shear information 236 and/or the motion information 238.

At step 1306, a function of the electronic device 150 based on the pressure-assisted gesture 250. In examples, the function of the electronic device may include any number of functions controlled by gestures, for example, hand writing, text editing, drawing, painting, User Interface (UI) interaction (e.g., object selection, menu selection, copying and pasting, drag-and drop functions etc. ) mode switching, user identification, encryption, among others.

While described primarily in the context of handwriting recognition, various other applications that require stroke generation and character recognition may benefit from some or all aspects of the present disclosure. Some examples include: drawing or painting, ubiquitous User Interface (UI) interactions and augmented or virtual reality, among others.

Various embodiments of the present disclosure having been thus described in detail by way of example, it will be apparent to those skilled in the art that variations and modifications may be made without departing from the disclosure. The disclosure includes all such variations and modifications as fall within the scope of the appended claims.

Although the present disclosure describes methods and processes with steps in a certain order, one or more steps of the methods and processes may be omitted or altered as appropriate. One or more steps may take place in an order other than that in which they are described, as appropriate.

Although the present disclosure is described, at least in part, in terms of methods, a person of ordinary skill in the art will understand that the present disclosure is also directed to the various components for performing at least some of the aspects and features of the described methods, be it by way of hardware components, software or any combination of the two. Accordingly, the technical solution of the present disclosure may be embodied in the form of a software product. A suitable software product may be stored in a pre-recorded storage device or other similar non-volatile or non-transitory computer readable medium, including DVDs, CD-ROMs, USB flash disk, a removable hard disk, or other storage media, for example. The software product includes instructions tangibly stored thereon that enable a processing device (e.g., a personal computer, a server, or a network device) to execute examples of the methods disclosed herein. The machine-executable instructions may be in the form of code sequences, configuration in-formation, or other data, which, when executed, cause a machine (e.g., a processor or other processing device) to perform steps in a method according to examples of the present disclosure.

The present disclosure may be embodied in other specific forms without departing from the subject matter of the claims. The described example embodiments are to be considered in all respects as being only illustrative and not restrictive. Selected features from one or more of the above-described embodiments may be combined to create alternative embodiments not explicitly described, features suitable for such combinations being understood within the scope of this disclosure.

All values and sub-ranges within disclosed ranges are also disclosed. Also, although the systems, devices and processes disclosed and shown herein may comprise a specific number of elements/components, the systems, devices and assemblies could be modified to include additional or fewer of such elements/components. For example, although any of the elements/components disclosed may be referenced as being singular, the embodiments disclosed herein could be modified to include a plurality of such elements/components. The subject matter described herein intends to cover and embrace all suitable changes in technology.

Claims

A computer implemented method for interacting with an electronic device having a touch sensitive surface configured to detect a stroke generation gesture on the touch sensitive surface, the method comprising:

receiving a rich stroke information corresponding to the stroke generation gesture, the rich stroke information including a positional information and a pressure information;

computing, by a pressure-gesture recognizer using the positional information and the pressure information: a pressure-assisted gesture; and

controlling a function of the electronic device based on the pressure-assisted gesture.
The method of claim 1, wherein the touch sensitive surface is a capacitive touch sensitive surface and the stroke generation gesture represents a movement of a tip portion of a digital pen device in contact with the capacitive touch sensitive surface.
The method of claim 1 or claim 2, wherein the positional information comprises a position profile including set of coordinates corresponding to points of contact of the digital pen device on the touch sensitive surface in response to the stroke generation.
The method of any one of claims 1 to 3, wherein the pressure information comprises a pressure profile describing a time-varying measure of a magnitude of pressure applied by the digital pen device to the touch sensitive surface in response to the stroke generation.
The method of any one of claims 1-4, wherein the rich stroke information includes a force information comprising a first tangential force and a second tangential force applied by the digital pen device to the touch sensitive surface in response to the stroke generation gesture, and the pressure-assisted gesture is further computed based on the force information.
The method of any one of claims 1 to 5, wherein the rich stroke information includes a pen motion information describing an orientation of the digital pen device in response to the stroke generation gesture, the motion information generated by a motion sensor coupled to the digital pen device, and the pressure-assisted gesture is further computed based on the motion information.
The method of any one of claims 1 to 6, wherein generating the pressure-assisted gesture comprises:

preprocessing the rich stroke information by resampling, rotating and scaling the positional information associated with the stroke generation gesture;

inputting the preprocessed rich stroke information into a Euclidean-distance based stroke recognizer to compare the preprocessed rich stroke information to a plurality of stored templates;

computing an average Euclidean distance between the preprocessed rich stroke information and each of the plurality of stored templates; and

determining a best match template of the plurality of stored templates corresponding to a minimum average Euclidean distance.
The method of any one of claims 1 to 7, wherein generating the pressure-assisted gesture comprises:

preprocessing the rich stroke information using a dynamic time warping method to determine a weighted DTW distance associated with the stroke generation gesture;

inputting the preprocessed rich stroke information into a Euclidean-distance based stroke recognizer to compare the preprocessed rich stroke information to a plurality of stored templates;

computing an average Euclidean distance between the preprocessed rich stroke information and each of the plurality of stored templates; and

determining a best match template of the plurality of stored templates corresponding to a minimum average Euclidean distance.
The method of any one of claims 1 to 8, wherein generating the pressure-assisted gesture comprises:

preprocessing the rich stroke information to obtain directional shear forces associated with the stroke generation gesture;

inputting the preprocessed rich stroke information into a Euclidean-distance based stroke recognizer to compare the preprocessed rich stroke information to a plurality of stored templates;

computing an average Euclidean distance between the preprocessed rich stroke information and each of the plurality of stored templates; and

determining a best match template of the plurality of stored templates corresponding to a minimum average Euclidean distance.
The method of any one of claims 1 to 9, wherein generating the pressure-assisted gesture comprises:

computing, by a machine learning (ML) model using the rich stroke information, the pressure-assisted gesture.
The method of claim 10, further comprising training the ML model, wherein training the ML model comprises:

obtaining a training dataset comprising rich stroke information for a plurality of example stroke generation gestures;

computing a pressure profile for each one of the plurality of example stroke generation gestures, based on the rich stroke information; and

training the ML model based on the rich stroke information and the pressure profiles.
A digital pen device for interacting with a touch sensitive surface of an electronic device, the digital pen device comprising:

a flexible tip portion comprising a metal electrode for contacting the touch sensitive surface;

a processing unit configured to execute instructions to cause the device to:

generate a rich stroke information corresponding to a stroke generation gesture, the rich stroke information including a positional information and a pressure information.
The digital pen device of claim 12, wherein the touch sensitive surface is a capacitive touch sensitive surface and the stroke generation gesture represents a movement of the flexible tip portion in contact with the capacitive touch sensitive surface.
The digital pen device of claim 12 or claim 13, wherein the positional information comprises a position profile including set of coordinates corresponding to points of contact of the flexible tip portion on the touch sensitive surface in response to the stroke generation.
The digital pen device of any one of claims 12 to 14, wherein the pressure information comprises a pressure profile describing a time-varying measure of a magnitude of pressure applied by the flexible tip portion to the touch sensitive surface in response to the stroke generation.
The digital pen device of any one of claims 12 to 15, wherein the rich stroke information includes a force information comprising a first tangential force and a second tangential force applied by the flexible tip portion to the touch sensitive surface in response to the stroke generation gesture.
The digital pen device of claim 16, wherein the force information is measured based on a bending of the flexible tip portion in contact with the touchscreen display.
The digital pen device of claim 12, further comprising:

a force sensor coupled to the flexible tip portion for receiving a force input applied to the flexible tip portion in response to the stroke generation gesture, wherein the rich stroke information includes a force information comprising a first tangential force and a second tangential force applied by the flexible tip portion to the touch sensitive surface in response to the stroke generation gesture.
The digital pen device of claim 12, further comprising:

a motion sensor coupled to the digital pen device for receiving a motion input corresponding to a motion of the digital pen device in response to the stroke generation gesture; wherein the rich stroke information includes a motion information comprising an orientation of the digital pen device in response to the stroke generation gesture.
A non-transitory computer-readable medium storing instructions executable by a processing unit of a computing system to cause the system to:

receive a rich stroke information corresponding to a stroke generation gesture, the rich stroke information including a positional information and a pressure information;

compute, by a pressure-gesture recognizer using the positional information and the pressure information: a pressure-assisted gesture; and

control a function of the electronic device based on the pressure-assisted gesture.