Back to Finger-Writing: Fingertip Writing Technology Based on Pressure Sensing

Received December 19, 2019, accepted February 1, 2020, date of publication February 12, 2020, date of current version February 28, 2020. Digital Object Identifier 10.1109/ACCESS.2020.2973378 Back to Finger-Writing: Fingertip Writing Technology Based on Pressure Sensing GADDI BLUMROSEN 1,2 , KATSUYUKI SAKUMA 1 , (Senior Member, IEEE), JOHN JEREMY RICE1 , (Member, IEEE), AND JOHN KNICKERBOCKER1 1 IBM Thomas J. Watson Research Center, Yorktown Heights, NY 10598, USA of Engineering, Bar Ilan University, Ramat Gan 5290002, Israel 2 Faculty Corresponding author: Katsuyuki Sakuma (ksakuma@us.ibm.com) (Gaddi Blumrosen and Katsuyuki Sakuma contributed equally to this work.) ABSTRACT Handwriting was since the start of the history, a higher expression of human skills, and was used for documentation of experiences, and for communication. Existing writing technology require a writing tool, like a pen, and a dedicated writing surface, like paper, or more recently an electronic tablet. These accessories of writing, of writing tool and service, are not available in many daily life situations. Furthermore, the writing accessories, are not natural, in many cases are not ergonomic, and thus can cause fatigue, and in extreme cases contribute to muscular and neurological diseases. In this work, we suggest to step back in history and step forward in technology, and to create, for the first time, an alternate writing solution without any accessories, using one own finger as writing tool, and write on almost any surface. For this, we used directional pressure sensors attached to the fingernail. Changes in the pressure induced on the fingertip in different directions while writing, are projected to the fingernail, and then assessed as a voltage pattern by the sensor. Decoding the pattern, can reveal symbols like letters, punctuations, and writing commands. In this paper, we describe the new pressure sensing modality and tailor processing methods. We tested the new technology on two subjects having different writing patterns while writing alphabet and sentences on different surfaces. We reached letter detection of over 80% while writing on a table, and the word detection rate, was near 70%, after applying the correction algorithm include language priors. The results of this work can revolutionize the way people write and communicate using more convenient, and more approachable, finger-tip writing. INDEX TERMS Handwriting recognition, human machine interface, natural language processing. I. INTRODUCTION Handwriting is an higher expression of human communication [1]. In beginning of human history, human started using their hands for with painting on caves’ walls [2]. Along time, writing tools like pencil, or pen were utilized to write over different surfaces like paper, board, and recently electrical tablet [3]. The existing handwriting techniques requires accessories of writing tool and dedicated writing surface, suffer from hand fatigue, and can contribute to the severity of muscular and neurological based diseases [4], and in extreme cases for an injury. This motivated recent attempts to find technologies to improve the writing experience. An optimal writing technique should be: ergonomic; can be used continuously for long time without fatigue and causing The associate editor coordinating the review of this manuscript and approving it for publication was Wenge Rong VOLUME 8, 2020 . injury; in any environment; and with minimal writing tools. There are two main ways to minimize the use of the writing accessories. One group of techniques built without dedicated writing surfaces, like paper or tablet, another one without writing tools like pens or pencils. To exclude the need from a dedicated writing surface, devices with sensory modules that can track the movement and transmit it wirelessly to a terminal can be used. An electronic wireless pen was suggested in [5], [6]. It is based on pressure measurements and/or motion of the pen, which is used as features for handwriting recognition in the computation terminal. Pen with accelerometers was suggested [7]. Another surface-less solution, is a fingerless glove that can implement a virtual keyboard for handwriting and was suggested in [8], and [9]. The angle at which the user’s finger bends at the proximal interphalangeal joint is used to decode a row of the keyboard. Discrimination between columns This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see http://creativecommons.org/licenses/by/4.0/ 35455 G. Blumrosen et al.: Back to Finger-Writing: Fingertip Writing Technology Based on Pressure Sensing operated by the same finger is achieved through an abduction/adduction sensor. To write without writing tool, some technologies use an electronic surface that is sensitive to finger touch. A Touch sensing display screen, can be on mobile devices like [10]. Writing letters on specific points on touch sensitive gloves was presented in [11]. The paper in [12], suggested to use a surface on the nail sensor to write different numbers. Both touch sensitive techniques, might be not comfortable writing in such device, not accessible, its performance are not always sufficient [13]. A tool-less and surface less writing techniques, can be based on assessing gestures that represent letters and other symbols. One way, is to attach to the finger magnetic sensors [14], or inertial sensors [15]. A ring, that include multiple sensor modalities was shown to implement a virtual standard QWERTY keyboard [16]. Sign gesture recognition using advanced temporal video image processing, and to decode sign gesture recognition using genetic algorithm, was developed in [17], [18]. All these techniques, require additional sensor modalities, are not fully seamlessly, need extensive training sessions as they do not exploit the natural way of writing. In this work, we suggest and demonstrate using a new sensor technology based on sensor pressure measurement of the index fingernail for finger-writing. The sensor technology fundamentals derived in IBM research labs [19]. This sensor enables using the natural intuitive hand movement seamlessly and thus is extremely ergonomic, uses the natural human way of writing from back in history, and can work on almost any available surface, in any environment. The pressure sensor can be implemented by a Strain-Gauge sensor (SG sensor) [20], or by Photoplethysmograph Fingernail Sensors [21]. The sensor is placed on the fingernail of the index finger. In the work in [22], a pressure sensor based on strain-gauge technology placed on the fingernail was introduced. It was shown, how the sensor is capable of measuring pressure point in real-time, can be used for assessing parameters like grip strength vital for medical diagnosis, and with machine learning tools, can also be used for coarse gesture recognition that can be used for enabling enhanced humanmachine interface. In the work in [23], we showed first feasibility of using the sensor for writing and recognizing basic shapes using only two pressure points. This paper extends the work in [22], and [23], and focus on using any pressure based sensor in task of writing full sentences in real-time in challenging conditions like different surfaces. The paper has the following contributions: 1) development of a paradigm for continuous finger-writing recognition using a fingernail pressure sensor. The paradigm enables using natural hand and finger movement and thus is extremely ergonomic, and emit the need from having writing accessories of writing tool like pen, and writing dedicated surface like paper. 2) Development of analytical tools that enable using the pressure sensor for writing and decoding of single symbol, word, and sentence in real-time. The methods are tailored to the unique characteristics of pressure 35456 FIGURE 1. A fingernail is used for writing (1.a) over different writing surfaces (1.b). The writing process over the surface induces pressure over the fingertip that is projected to the nail. A Pressure Sensor (PS) unit is attached to the finger nail to capture the pressure pattern. It is implemented by embedding few strain-gauge in different directions (1.c). The raw data is then transmitted wirelessly to a storage and processing unit (1.d). measurements, and differ from other methods like [16], where the analysis pipeline for accelerometers and gyroscopes was reported. 3) Showing first feasibility of the technology and comparing to accelerometer based only sensors. 4) Provide first validation of the new finger writing technology in different challenging every daily-life condition, like writing over different writing surfaces. The paper outline is as follows: in section II the sensor system and the sensor technology is described; in section III, the methods tailored to the sensor for real-time writing finger writing are derived; section IV, describe the experiment setup; the results are shown and discussed in section V; and in section VI a summary of the work, and future directions are given. II. SENSOR DESCRIPTION The system is based on sensing the pressure projected into the fingernail while writing on a surface. The sensor fundamental technology was derived in IBM research labs [19]. The pressure sensor can be implemented by a Strain-Gauge sensor (SG sensor) [20]. This sensor enables using the natural movement freely and thus is extremely ergonomic. The sensor is placed on the fingernail of the index finger, and doesn’t disturb the subject to perform his/her natural activities. When the subject is writing using his index fingertip, a pressure is induced on the nail, which results in deformation of the SG sensor. The deformation is translated to voltage-change data stream, which is transmitted wirelessly to a computation for decoding. The suggest system is composed of the following parts: a finger that is used for writing (a), a writing surface or medium (b), Pressure Sensor (PS) unit (c), and storage and processing unit (d). Figure 1 describes the system. The PS unit is attached to the fingernail (Fig 1.a). The surface for writing (Fig 1.b) can be any surface that induce changes in pressure on the nail, when writing with the fingertip, e.g. 2-D surface like table, but can also be non-rigid VOLUME 8, 2020 G. Blumrosen et al.: Back to Finger-Writing: Fingertip Writing Technology Based on Pressure Sensing FIGURE 2. Processing data flow in finger-writing recognition. The processing consist of the flowing stages: a) pre-processing of the PS raw-data, which include excluding of sensor bias, filtering, scaling, and interpolatio; b) detection of start and end boundaries of each written shapes; c) shape feature extraction; d) symbol recognition based on the feature shape, that include also multi-shape symbol, and is based on individual and public trained symbol data bases; and e) word and sentence detection using Natural Language Processing (NLP) exploiting language priors, dictionary. surface like cloth, or even the air (movement of the body parts in the air, can induce changes in the pressure of the nail). The sensing unit is shown in Fig. 1.c. The PS unit can be implemented by different technologies that can assess pressure in different directions. In this work, we choose to implement the PS by Strain-gauge technology (Fig 1.c1); additional sensor like accelerometer can be aggregated (Fig 1.c2); the physical pressure measurements, are digitized, and pre-processed in the pre-processing unit (Fig 1.c3), and then sent wirelessly for further processing to decode the finger writing and store the results (Fig. 1.d). The sensor is placed on the nail. When the fingertip interacts with surfaces, a unique pattern of pressure is induced on fingertip and the pressure pattern is projected to the fingernail (Fig. 1.a). The figure demonstrate the pressure map, as was obtained by thermal mapping on the nail, when the pressure is induced on the center. This pressure directionality pattern can be used similar to [24], for evaluating the direction and intensity of movement of the finger over time and enable extracting informative features to decode the finger writing. III. METHODS The methods derived in this work, form the baseline for Finger-Writing (FW) recognition using directional pressure sensor. The subject writes and induces pressure on the surface, which translates to deformation of the nailin different directions. When the subject stop writing, the pressure is relaxed, and go to steady state values. The pressure pattern VOLUME 8, 2020 induced while writing a shape on the surface is correlated to the shapepattern. The shapes are than aggregated and recognized as symbols of letters, or numbers, or punctuation, or commands, or breaks between words. The symbols and their probabilities are used to higher level meaning of words and the complete sentence. The FW method is composed of the flowing stages: a) pre-processing of the PS raw-data; b) detection start and of writing actions of writing the shapes; c) shape feature extraction; d) symbol recognition; and e) sentence detection. Figure 2 describes the processing data flow. A. PROBLEM FORMULATION A symbol is defined as the minimal information unit that convey meaning, and can be a letter, punctuation, mathematical expression, general writing commands, or a keyboard like commands like break between words, or deletion of a letter [25]. Let define the k’th symbol, as gk , where k=K gk ∈ {gk }k=1 s . A sentence is then composed from a series of J symbols: s = g1 g2 . . . gj . . . gJ (1) The criterion for FW recognition to find from the sensor’s raw data the sentence that was written is:   ŝ[t,t+T0 ] = s[t,t+T0 ],î =argmax i P s[t,t+T0 ],i εt . . . εt+T0 , (2) where εt , εt+1 , εt+T0 , are the PS’s inputs over period of time [t, t + T0 ], corresponding to different location and finger 35457 G. Blumrosen et al.: Back to Finger-Writing: Fingertip Writing Technology Based on Pressure Sensing orientation, and s[t,t+T0 ],i , is the i’th hypothesis of a possible sentence from almost infinite possibilities of symbol permutations. In case there are more sensor modalities, the recognition depends also on the other sensor data, which is assumed here, without loss of generality, as accelerometer data, denoted by, at , at+1 . . . at+T0 . The recognition criterion than becomes:   ŝ[t,t+T0 ] = argmax i P s[t,t+T0 ],i εt , . . . εt+T0 , at , . . . at+T0 (3) To solve the criterion, we divide the solution to four stages: 1) detect from the raw data, shape start and end time for each pressure/relaxation session; 2) extract shape features; 3) recognize symbol’s set of shapes and then recognize the symbol, like letters, punctuation, commands and breaks between words; and 4) decode the higher level meaning of words and the complete sentence. B. PREPROCESSING The N SG sensor (SGS) outputs, denoted by: ε1t , ε2t , . . . εNt , reflect the change of the voltage due to the nail deformation from the finger writing operation. These outputs can be estimated as follows [20]: Dti = KE ti εit (4) where i moves between 1 to N , Dti is the deformation, Eit is the transfer constant of pressure to voltage, and K is a normalization constant that is a function of the temperature. The deformation of the nail is spatial, and varies with the nail surface, and over time. Normalization of the pressure measurements can then be seen as a measure to the nail deformation up to the constant (KE ti ). The pre-processing process consist of the following stages: exclusion of the changing voltage bias, filtering out of artifacts, mitigate for missing samples, scaling the pressure signal, rotation of axes, and dimensionality reduction in case of multiple pressure points and when interpretable directions are needed. To remove the DC effect, a high pass filtering of typically 0.1 Hz is used to exclude very slow movements and biases that are not likely to be part of the writing. To mitigate over missing samples, the missing samples are interpolated. Distorted samples are then filtered out by using low-pass filtering. The nail sensor can suffer from changing bias, due to temperature change, from undesired induction in the circuit [26], or from battery aging. This bias can be removed estimating continuously the bias, and excluding it using Kalman filtering similar to [27]. Scaling of the input is applied to standardize the features and improve their tolerance to different subjects’ finger pressure profile, different pressure surfaces, and by this to improve the overall classification results [28]. Dimensionality reduction can be also used when the number of SGS is high. The signal after pre-processing, is a signed temporal vector for each SGs. It can be characterized by its amplitude, and 35458 polarity. The polarity, reflects the direction of the induced voltage, and can be estimated in calibration phase. The SG’s amplitude can be used to estimate the intensity of the pressure, while the sensor polarity, can be used to estimate the direction of writing. The sensor polarity, is unique to the subject, and is related to SGs location, nail topology and sensor properties. For the additional inputs (like accelerometers), a similar pre-processing, based on characteristics of the signals, should be obtained. C. DETECT SHAPE BOUNDRIES A precise detection of symbol start and end times is required in FW recognition, similar to handwriting recognition [29]. Unlike visual based writing recognition methods, the task for shape start and end of writing times, can’t be detected by spatial time discontinuity, but need to be derived by temporal measures or learning [7]. When the subject writes with his fingernail, a pressure is induced on the surface, which is then projected to its fingernail and the measured pressure move from steady state level to active pressure values. Then when he finishes inducing pressure on the surface after writing the shape, the pressure goes back to its stead state condition, which is not necessarily the one with minimal pressure change of the sensor. This steady state pressure level can be considered as the reference point, or the ‘‘off’’ state, denoted as OFF. Deviation from this state, are defined as ‘‘on’’ state, denoted as ON. These states, are often referred to as loading and unloading states [30], and can help to detect start and end writing times [31]. The ways to measure the deviation from the reference state can be performed by edge detection algorithms, e.g. by thresholding the signal over the stead state level, and in case the PS have deformation together, by using the envelope, composed of average sum of the PS signal or on the PS components. The PS envelope can be defined as: q p 2 2 2 2 2 2 (5) r t = ε1t + ε2t . . . εNt = εt where εt , is the vector of PS measurements at instance time t, ε t = [ε t1 , ε2t , . . . εNt ]. Acommon way to detect shape start and end, is by thresholding the envelope, when compared to the reference t . More advanced methods, can use statistical OFF state, rref features of the envelope, like the Sobel edge detection algorithm [32]. We denote the start and end of the k’th shape as tsk , tek . D. EXTRACT SHAPE FEATURES Each shape, in the writing interval, has a typical pattern. Based on the pattern, features are extracted and fed continuously to a symbol classifier. This pattern can be used as a feature, or to extract physical interpretable features like pressure intensity, and shape duration. Spectral features like peak and median frequencies in [33] can be used. Features can be derived automatically by exploiting neural network [34]. VOLUME 8, 2020 G. Blumrosen et al.: Back to Finger-Writing: Fingertip Writing Technology Based on Pressure Sensing Since FW is unique in the sense it is not limited to use with memory-based surfaces like a tablet or paper, it lacks spatial information. Thus, the FW recognition is more complex, and it needs to be based more on temporal features, like start and end of a new shape, shape duration, delay between consecutive shapes, and the temporal pattern to try to compensate over the missing spatial information. Additional sensors can be used to provide spatial information that can be aggregated to the PS’s based features, and together provide spatial-temporal information. We denote the k’th feature set as: k Fsh = {F k1 , F k2 , . . .}, (6) where k moves between 1, to K0 , and indicate the shapes’ index in the sentence writing time, [t, t + T 0 ]. E. SYMBOL RECOGNITION In many cases, symbols like capital letters are combination of a few shapes, and then aggregated to a symbol. The aggregation uses statistical relationships between the shapes and learning on prior-knowledge related to the shapes [23]. A shape aggregation criterion for finding the start and end of the j’th symbol is:   h j ji n o k<K0 ts ,te k j j (7) Fsh , t s , te , ĵ = argmax j P s k>1 where j moves between 1, and the number of possible symbols, Nsym . This criterion, can be solved by deploying clustering techniques like the one used in Sonar and Radar systems [35]. The j j symbol start and end time, ts , te can be found using statistical methods that combine the best decoded hypothesis from a pool of recognizers used in speech recognition [34], and in mathematical expression recognition [7]. The hypothesis ĵ, for the shapes related to the symbols, can be used to form the j symbol feature set Fsy . After start and end symbol detection, the symbol classification recognition is reduced to simpler pattern matching problem estimation:   h j ji t ,t j , (8) ĵ[t,t+K0 ] = argmax j P s s e Fsy n K o K +1 K j where Fsy = Fshj,s , Fshj,s . . . Fshj,e are the features of the j K j K j’th symbol, and ts = ts j,s , and te = te j,e . Without any loss of generality, we choose for this study, set of features relate to the writing pattern, and are complementary set that emphasize non-pattern characteristics like writing duration and intensity. The following features were examined for each SG separately for the non-pattern features: Letter writing duration, τ , maximal pressure, Amax statistical characteristics of pressure: mean, standard deviation, skewness, kurtosis (Amean , τA , σA , S, K ); pattern properties features: Number of positive and negative peaks, and their relative location (NPp , NNp , PPp , PNp ); and Spectral features of peak and median frequencies, energy content at low, medium, and high frequencies (fmax , fmed , EFl , EFm , EFh ). VOLUME 8, 2020 In case where a symbol is composed of one shape, Kj,s = Kj,e , and there is no need for shape aggregation. For more accurate estimation, the criterion in (8) can be conditioned on prior knowledge available on fingertip physiological movement constraints, like typical values, maximal and minimal features values similar to [36]. The recognition of the symbol in (8) is a classification problem and machine learning techniques like handwriting can be used [6]. A training phase can incorporate the prior-knowledge related to the symbol. The classifier output is a vector that consists of the confidence probability of each symbol that can be fed to the word/sentence decoding algorithm. The j’th decoded symbols in a sentence, is denoted by gĵ , and its confidence j vector is psy . In cases where recognizing the writing of one subject, a relatively small change in the pattern between consecutive writings of the letters (symbols) is expected, then methods like Dynamic Time Wrapping (DTW), can be utilized like in speech recognition [37]. The symbols pattern, with different velocities’ hypothesis, can be then correlated to a dictionary of patterns of different symbols patterns of the k=K subject, {gk }k=1 s . When the pattern vary among different subjects, more advanced classifiers like deep learning, should be applied in future. Design of commands in FP, is challenging task. In general, symbols like letters and punctuations emerge from perceptual and motor models [38], and thus are assumed to have low spatial correlation. But in some punctuation symbols that involve low and short pressure, like dot, or comma, they are less separated, and are more likely to be erroneous. For commands, like space, special FP commands can be designed that either have spatial difference, or induce change in temporal pattern, like repeating short localized pressure, resembling a point, few times, in different speeds. To choose commands with low distance tools from space-time coding like in [39] can be used. Space is a special command, since it can be evaluated, by the delay between the writing of different words, which is natural, since the time between words in handwriting is also longer than between words, as the pen needs to move a longer distance. The distribution of the delay between words can be assumed to have a Gamma distribution [40]. Therefore, the likelihood of having a space between two consecutive symbols is: pjsy,space = Ŵ(tsj+1 − tej ) (9) The parameters of the distributions can be found empirically, and are subject dependent. To find the end of one word, or the start of new (keyboard space) a boundary between the distributions should be found, that minimizes false detection. Usually the subject is expected to induce longer delay after the last letter in a word, following the natural way of writing, but even if not, through training and visual feedback the false detection can be minimized. 35459 G. Blumrosen et al.: Back to Finger-Writing: Fingertip Writing Technology Based on Pressure Sensing F. SENTENCE DECODING The symbols’ confidence values for each symbol over time, together with language prior knowledge, as described in Fig. 2, are fed to classifier to decode a full sentence. The sentence decoding criterion, based on the raw sensor data in (3), is reduced to the following criterion:   ŝ[t,t+K0 ] = argmax j P g1 . . . gj . . . gJ p1sy , . . . pJsy , PL , (10) where PL , is language prior, trained on data set. The language priors, that is commonly used in handwriting and speech recording is sometimes refer to as priors of Natural Language Processing (NLP) [41]. It includes the likelihood of letter sequence, which is related to human vocal properties and limitations, word dictionary, relationship between the word, as can be determined from higher level processing and context. This relationship can be extracted by training the model over a large amount of data with many examples and large computation resources using advanced machine learning techniques, such as deep learning. A sub-optimal approach that requires reduced level of trained data, and less cumbersome computations, is to use the traditional approach of first detecting the words using punctuations symbols, and subsequently use for each word, a Hidden Markov Model (HMM) [42], that include the letter probability and their relationship over time. Dictionary correction can be applied as post-processing stage. The accuracy of this algorithm depends on the symbol detection accuracy, the accuracy of transition between letters, and on the accuracy of the break probabilities. Thus, for each word, the HMM, solve in Maximum Likelihood (ML) optimal manner the decoding of each word, conditioned on the features of the letters of the word. Upper layers corrections, can be applied after the stage, using language based using query spelling suggestions [43]. IV. EXPERIMENT SETUP A. EXPERIMENT SETUP DESCRIPTION The experiment setup is composed of a PS sensor equipped with wireless capability, a computer to process the data and display the results, as described in section II, and is shown in Fig 3.a. The sensor unit is composed of three SG sensors and 3-D accelerometers, amplifiers, analog to digital converter (ADC), micro-controller for initial processing, a wireless data transmission and a battery. The sampling frequency was set to 50 Hz. The A/D output signal range is 0 to 2047, where the maximum voltage of ADC is 1.87V. An offline power control was set that the maximal voltage range will fit the strongest deformation of the SG. A 3-D acceleration system on the nail was used as a reference system to the pressure one. Before the experiment, the sensor was attached to the index fingernail, using a nail glue. The directionality of the sensor was then tested by rolling the finger on a surface, to verify the coupling of the sensor to the nail and to assess its directionality pattern. The driver and the processing methods were implemented using MATLAB (Matlab.com, 2016b). 35460 FIGURE 3. Processing data flow in fingertip- The experimental setup (Fig. 3a). Figure 3b, shows the three writing surfaces used in the experiment, table, cloth, and the subject’s hand. The experiment for evaluation of the method was performed with two subjects, which English is their second language, and thus their writing style is different. They performed writing on a table, cloth, and on their hand, as demonstrated in figure 3.b. B. EXPERIMENT SETS The aim of the experiment sets was to characterize the PS sensor signals, provide feasibility for the methods for decoding the FW, and produce first order performance evaluation of the new technology. For this, we designed an experiment set with two subjects that write sentences, in natural way as possible, on the different surfaces. We examined the inter-subject variability in a session, and the intra-subject variability. FW, similar to handwriting, has different characteristics pattern that changes from subject to subject and over time, and writing surface. There were two main sets. One was built to examine the performance of writing on different surfaces (table, cloth, and hand), when trained on others (table). The other one was to test the suggested encoding of full sentences, when trained on only two repetition of the alphabet (one shout training), which is short enough to be used in daily life system after the deployment of the sensor. For both sets, the training was writing the English alphabet with two repetitions on the table. The first test set was writing on three surfaces (table, cloth, and hand) twice, the entire alphabet. In the second set, each subject wrote same 9 sentences (that were randomly chosen from the start of the book ‘‘Moby Dick’’) on the table, with varying word length from 2 to 9 letters, in total of 33 words. V. RESULTS A. SIGNAL PRE-PROCESSING Before the start of the experiment, the sensor was tested to verify that the sensor was attached properly, and to assess the specific fingernail topology, similar to what is used in other sensing technologies that require high level coupling like Electroencephalography (EEG) sensor modality [44]. The absolute directionality response of the sensor, and its polarity, that can be used to estimate writing intensity and direction, are obtained by rolling the finger from right to left (roll) with VOLUME 8, 2020 G. Blumrosen et al.: Back to Finger-Writing: Fingertip Writing Technology Based on Pressure Sensing FIGURE 4. Directionality and absolute response of the SGs from finger rolling test demonstrated by the first subject. The pressure sensor three SGs correspond differently in their amplitude and polarity (sign of the voltage) to different pressure directions. These patterns, are unique to the subjec’s nail topology and sensor properties can be used to estimate the writing direction, and the pressure intensity. different orientation. Figure 4, shows the amplitude and polarity (sign) of the rolling test, performed by the second subject. The separation between the SGs sensors, which govern the sensor directionality, is significant in finger rolling angles of over 20 degrees. The correlation coefficient between left, and center SGs is 0.5, right and center is negative 0.483, and right and left, of negative 0.339. This indicated a small coupling between left and center SGs, and more dominant negative polarity of the right SG, which is related to the fingernail morphology at the SG’s location. The SG signal is pre-processed to mitigate over missing samples, exclude artifacts, and compensate for bias. Figure 5 shows the signal in writing small letters alphabet. The miss sample rate was less than 0.5 percent, and the missing samples were distributed along the entire training session, and thus can be mitigated by interpolation similar to [45]. From the figure, it is seen that the SGs captures around half of the voltage dynamic range in regular writing operation, which will still support double force induced on the surface. The voltage reference points changed between each SG by less than 10 percent of the full dynamic range. The signal as seen in Fig 5.a, suffers from slow frequency drift over time, where there is a change of the OFF reference point from around in the beginning of the session to decreases by around 0.03 V in the end of the writing session after around 50 second. A small-scale variable random change can induce changing bias. For the removal of the slow bias, a high-pass filter of 0.1 Hz was applied. For the continuous drift, a Kalman filter that continuously estimate and subtract the bias by using other sensor measurements similar to [27] can be applied. The data is then normalized according to the strongest pressure value in the session. The data before, and after the filtering, is shown in (Fig. 5.b). B. SYMBOL BOUNDARIES DETECTION Accurate shape and symbol boundary detection affects the accuracy estimation of the FP writing. In the case where a symbol is a multi-shape symbol, like in mathematical expression, or in case of writing capital letters not in cursive writing, the shape boundaries should be decoded, prior to symbol’s shape aggregation and calculation of symbol start and stop. In case of one-shape symbols, or a symbol that is composed from multiple shapes, but its writing is performed continuously where the finger does is in OFF stage for only fraction of second, like in cursive writing, the symbol start/end time can be estimated directly. Since in FP writing process tries to be with minimal limitation, and mimic the natural way of handwriting, the detection algorithm for shape or symbols, can exploit prior knowledge in handwriting, like typical delays between the symbols, symbol duration, and the typical pressure level induced in FIGURE 5. Signal of writing alphabet in small letters, before and after pre-processing. The pre-processing includes interpolation to mitigate over possible missing samples, high pass filter to exclude bias, and normalization of the signal. VOLUME 8, 2020 35461 G. Blumrosen et al.: Back to Finger-Writing: Fingertip Writing Technology Based on Pressure Sensing FIGURE 6. Signal of writing alphabet in small letters, before and after pre-processing. Punctuations of ‘‘?’’, and ‘‘!’’ symbols are shown in (6.a), and their shape seems quite separated. For other punctuations like dot, and other writing commands like space, or delete, we suggest using dedicated gestures performed twice as shown (6.b): bottom up, up bottom, left right, right left (comma, period, delete, space). The entire alphabet writing as used for training, and its start and end are detected automatically, are shown in (6.c). The envelope used for detection of OFF and ON states, is shown below. the FP writing. This is used in setting the envelope detection thresholds. The ON state was determined, when the letter envelop in (5), went down from the peak to 30 percent above the OFF reference level. Figure 6.a, and 6.b, shows an example of writing punctuations and commands symbols. The punctuations were of ‘‘?’’, and ‘‘!’’. For other punctuations like dot, and other writing commands like space, or delete, we suggest using dedicated gestures, as shown in Fig. 6.b. The commands are performed twice, continuously, bottom up, up bottom, left right, right left (comma, period, delete, space). Figure 6.c, shows the entire alphabet writing that was used for training, and its start and end are detected automatically At the bottom of Fig. 6.c, the signal envelope (3) is shown, and the limits between OFF and ON states. C. SIGNAL REPRESENTATION/ FEATURE EXTRACTION Each shape and symbol is characterized by a typical pattern, and a set of temporal, spectral, and kinematic features. To characterize the signals pattern, we normalize their power, and wrap them into a 256 length vector. The normalization, helps to emphasize FP’s writing pattern differences that are invariant of writing speed, and is similar to what is used in machine learning based techniques [28]. Features matching algorithms can then be applied on the pattern waveform. Neural network based pattern matching techniques required massive data sets, and are less interruptible [46]. In addition, since the main focus of this initial study is to characterize the signal in relation to the physical sensor features, we used as features the coarse signal pattern, in addition to fundamental common features commonly used in handwriting recognition and have interpretable physical meaning [29]. 35462 Figure 7 shows the normalized pattern of the three SGs while writing English alphabet on the table during training for the first, and second subjects. The left, central, and right SGs patterns are in blue, red, and green colors, and the inter-session variability (2 repetitions) is characterized by the area around the line. The letters start and end were found according to (3). The direction of the FP writing over time can be decoded by the directionality pattern. The polarity of the central and left SGs is positive, while the right SG, has dual polarity, which can be related to the relation between the nail morphology and the sensor location. It can be seen from the figure, that the inter-session variability is much lower than the inter-subject variability. It is explained by subjects’ different writing style, that include writing same letters sometimes in different directionality (like writing the letter l, bottom up, or up bottom), different pressure (individual strength, or different nail shape that project different pressures), and different timing. This implies that subject specific patterns, or massive data from many diverse subjects should be used to reflect these kind of variations for training in future. Writing of the letters c, and m, at different writing surfaces (table, cloth, and hand), for the two subjects is shown in Fig. 8. The letters c is one of the simplest letters that usually has lower subject variability (writing half circle counter clock wise) in its writing compared to other letters [23]; therefore, it can be used as a reference letter. From the patterns, it seems that the letter is written in similar way in the different experiment conditions (surfaces) for the different users, but with higher inter-subject variability due to different directionality pattern between the two subjects. The letter m is more complex in writing, and has higher variability in its writing, as it contains three lines, and 2 half circles. For the two subjects, VOLUME 8, 2020 G. Blumrosen et al.: Back to Finger-Writing: Fingertip Writing Technology Based on Pressure Sensing FIGURE 7. Normalized writing patterns of alphabet for writing in the table for the first and second subjects. The inter-session variability (2 repetitions), is characterized by the area around the line. The inter session variability is much lower for most letters, than the inter-subject variability, due to each subject writing style, that include different letters directionality (writing from left to right, vs. writing right to left), pressure (individual strength, or different nail shape that project different pressures) and timing. Thus, subject specific patterns, or massive data from many diverse subjects, should be used for training. FIGURE 8. The pattern change between subjects and writing surfaces for the letters c, and m. The differences between the experiment conditions (surfaces) are much lower than between the subjects. This indicates that for accurate results, subject specific training is preferred, and that it is possible for each subject to recognize letter, when trained in other surface. at least two peaks (positive, or negative) can be recognized, which suits the nature of writing similar shapes. The directionality of the PS can be used to assess the direction of the FP movement. For instance, the pressure on the fingertip while going down, and releasing the pressure while going up, can be seen on the central SG, and in some of the SGs that are coupled. The direction of movement can be assessed by the magnitude of the un-normalized SG amplitudes in the calibration as shown in Fig. 1. For instance, the right SG is more dominant than the left one in most letters, due to the nature of writing from right to left. This value of magnitude is one of the features that are used in addition to the pattern in the symbol recognition. Writing the letter on a cloth, is characterized by higher inter-session variability, and less smoothness compared to the smoother surfaces of the hand and the table. Writing over the VOLUME 8, 2020 hand, is characterized by the highest inter-session variability, which can be explained by the different curves on the hand, and different start writing points. For the two letters, the intersubject variability is higher than the inter-session variability, which indicate on the need to tailor calibration for each subject separately, or use massive data side, that capture all the variations in different letters writing. The average inter-subject letter variability is function of the subject, his/her technique of writing, and of the writing surfaces. Table 1. shows the inter-subject variability (between experiment repetitions) in different surfaces, calculated as the mean distance in the feature space. The table inter-subject variability is the lowest, which can be explained by the relatively smooth and more certain surface of the table. The other two surfaces tend to be more challenging for writing, with more varying surface in one hand, which result 35463 G. Blumrosen et al.: Back to Finger-Writing: Fingertip Writing Technology Based on Pressure Sensing FIGURE 9. Normalized features values in different surfaces for all letters for the central pressure sensor for the two subjects. The deviation of feature’ values and between the subjects is high. Some feature’ value are consistent between the subjects, like the maximal pressure, Amax (highest on the table compared to the hand and cloth), or the maximal frequency of letters, that is the lowest on the table, due to the smoothness of the table. TABLE 1. Inter-subject variability. in higher inter-subject variability. The inter-subject variability (variability between the subject, intra-subject variability) is much higher (>50% in average) than the inter-session variability of same subject, which can be explained by the difference in the sensor placement and the differences between the subjects’ nail morphology. Figure 9 shows the normalized mean and standard deviation of the features values in training data, averaged on all subjects. It seems that there is no consistency among subjects on the deviation of the features among the surfaces. The letter duration seems to deviate between the two subjects, may be due to the writing limitation of the experiment. The pressure power seems to be the highest on the table and on writing with on the cloth, for subject one, and highest on table, and lower in cloth and hand surfaces. This can be explained by the smoothness of the surface, and by the subject’s individual motor control. For example, when the surface is smoother (like the table) to have effective writing with efficient effect on the nail pressure, the subject needs less writing power. The maximal frequency of writing seems to be the lowest on the table and can again be explained by the smoothness of the writing surface. Applying F-test, to the features, shows that the four most significant features to recognize the letters in the training sequence in decreasing order, are: µA (the average pressure when writing the letter), Efm (the average frequency content of the signal between 1 to 3 Hz), τA (the duration of writing the symbol), σA , the standard deviation of the signal, which indicate on the smoothness of the writing. 35464 FIGURE 10. Recognition accuracy dependency on the writing surface of the PS (with DTW and DFA classifiers), in compare to accelerometer (with DFA). The data was trained on writing over the table. The DTW classifier, which exploit the pressure pattern only, seems to overperform the DFA classifier, which exploit statistical features related to the pattern, pressure strength, and duration. D. SYMBOL RECOGNITION Symbols can be letters, which are composed of shapes (in case of capital letters), single shape letters (small letters), punctuations, or writing commands like the one implemented in keyboard. In this work, we focus, without loss of generality, in recognizing single shape letters. Extension of the work to the other symbols can be achieved using methods like in [23]. For the symbol recognition, we used two classifiers that are traditionally used in handwriting recognition, Dynamical Time Wrapping (DTW) [47], and Discriminative Function Analysis (DFA) classifier [48]. The DTW works on the pattern while the other classifiers can work on the features, pattern, and combination of both. To reduce overfitting in the training phase, we added noise according to statistics of the measurements similar to [27], and performed on a features Principle Component Analysis (PCA) and kept the components that preserved 98% of the explained variance. For the training set we used for the DTW, the alphabet FP on the table, which due to the smoothness is ‘‘cleaner’’ pattern of the signal, and thus is natural choice to preserve the reference pattern. For the DFA classifier, the alphabet writing at different surfaces was used, to reflect the spread changes of the pattern in the different surfaces. The training error for the DTW, and DFA, was 9.9, and 3.8, and 9.9, and 3.8 percent, for the first, and second subjects, respectively. The letter detection rate for the multiple experiment sets is shown in Fig. 10. It shows the results of DTW, based on the pattern only applied on the PS’s data; DFA classifier applied on the PS’s data, based on pattern after dimensionality reduction using PCA to 20, in addition to the 7 significant features from each PS point (left, center, and right), and DFA applied similarly on the reference acceleration data. The accuracy of the training set was high for all classifiers, and for both PS and acceleration measurements. The accuracy of the DTW was higher for the PS’s data, which indicate that the information in the pattern is critical for high recognition rate. The test set using the table as reference, was the highest, as expected, since the classifier was trained on the table, and the table is the smoothest surface, and therefore, is likely to maintain VOLUME 8, 2020 G. Blumrosen et al.: Back to Finger-Writing: Fingertip Writing Technology Based on Pressure Sensing FIGURE 11. Histogram of delays between consecutive writing. The separation between words and letters is determined by threshold that minimizes the word start false detection, which depends on the Gamma approximation accuracy. The threshold was found to be 0.65, and 0.85 seconds for the first and second subjects respectively. consistency with the patterns over time. The accuracy of writing on the cloth, was still high, as beside the increase friction, it doesn’t have unexpected curves, like the hand, that has the lowest performance. Training the FP writing on the cloth, and the hand, might come with higher detection rate for these surfaces, but will down perform the performance on the table, which is around 80%. This shows a tradeoff between high detection performance and compatibility to different surfaces. The acceleration reference, gave performance of around 20%, which is less than third of the accuracy using the pressure sensor, but around 5 times higher than the random selection of 3.85% (shown in black line). To detect word’s start/end times, a dedicated symbol can be used, that can be for example matched to the keyboard symbol set. FP writing can exploit the delay between letters and words and impose way of writing that is close to natural writing style, but with the awareness of the subject to have longer break between words than compared to between letters. This is equivalent to the increased duration in handwriting, when moving the pen when start writing new word, compared to the one when writing small letter. Figure 11 shows the histogram of delays of between writing letters vs. the one between writing words. The curves (red and blue), show the Gamma distribution fitting for the letter and words. The separation between words can be determined by threshold that minimizes the word start false detection according to the distribution tail, which depends on the Gamma approximation accuracy. The threshold was found to be 0.65, and 0.85 seconds for the first subject, second subject, respectively. Using this threshold, with one misdetection of word start, there were 4 (4.4%) and 1 (1.1%) false detection of word start (as part of the word). The difference in the distributions, and thresholds between subjects is due to difference in writing characteristics like velocity, and the way the subject perform the OFF phase, when the subject raise his/her finger, between writing the letters. Thus, more training in FP, real-time visual feedback, and methods that know to correct the spaces between words according to the word and sentence content, are expected to improve the word start and end recognition. E. SENTENCE DECODING Figure 12 demonstrate an example for a sentence data analysis flow for FP writing of the first subject. The sentence, is ‘‘never mind how long ago precisely’’ taken from the book Moby Dick. Figure 12.a shows the pressure signals after post-processing and applying the symbol boundaries detection algorithm. Features are extracted and fed to DWT based classifier, which output the likelihood of each FIGURE 12. Histogram Recognition of FW example for the sentence from Moby Dick, ‘‘never mind how long ago precisely’’. The letters boundaries are first detected. Then features are extracted and fed to classifier. Then are decoded for small letters where each word recognized by its pattern. Unlike the capital letters that was composed from symbols representing shapes, in small letters, the symbol can represent the all small letter. VOLUME 8, 2020 35465 G. Blumrosen et al.: Back to Finger-Writing: Fingertip Writing Technology Based on Pressure Sensing TABLE 2. Word recognition success rate statistics. j letter, psy , in the confidence matrix in Fig. 12.b. The right column shows the probability of the break between words, j psy,space , denoted in the figure as Pb . To decode a letter, we look on the maximal probability of the classifier in a row. Then, an internal HMM model is applied, that output the two maximal hypotheses. We apply the following efficient algorithm that exploit the first, second, and the spelling corrections, used as prior knowledge of the language. If the first HMM hypotheses is a valid dictionary word, then we choose this word. If not, we apply the dictionary on the second HMM hypothesis, if it is part a valid dictionary word, then we choose this word. If not, we choose the dictionary correction of the first HMM hypothesis. If there is no correction to the word, which is likely with names that are not part of the dictionary, then we keep the first HMM hypothesis. The sentence is then fed in post processing stage, to an algorithm using query spelling suggestions like NLP in [43], to improve the results accuracy, as seen in the bottom of Fig. 12.C. We compared three different methods for word recognition: 1) HMM hypothesis without applying any prior letter knowledge; 2) with word spelling correction applied on first two HMM outputs, which is example of applying word prior knowledge; and 3) with applying on the HMM with prior knowledge outputs prior knowledge related to the sentence (NLP), using google suggestion tool. After first HMM hypothesis, the word detection success rate were, 49.3, and 45.5 percent (compared to around 3 percent for random selection). The detection rate of words without or with only one spelling errors based on the first HMM hypothesis, were 84.8, and 75.7 percent. After applying the suggested dictionary correction based algorithm from above, the success rate increased to 63.6, and 64.2 percent. This indicates that only part of the one letter level errors could be mitigated by using the prior knowledge about the words. After applying NLP knowledge, the relationship between words in sentence, and the content, the percentage increased to 69.7, and 66.0, for the first and second subjects receptively. Table 2 summarizes the word performance. VI. CONCLUSION AND FUTURE WORK This work describes for the first time, systems and methods that has the potential to enable natural writing with a finger on almost any available surfaces without need of writing accessories. In this work, the new concept was described, and processing methods were tailored to the nail sensor. The new technology was validated by two subjects with 35466 different writing style patterns, different nail morphology, on different surfaces. The accuracy of the letter detection on table was over 80%, and the word detection rate, was near 70%, after applying the full correction algorithm include language priors. Adding more pressure points, extending the data base size to reflect all the settled writing variations, and aggregating information from other sensors, can improve the results. The results of this work can also assist in detection of abnormal pattern of writing and assist in making writing more accessible to new populations throughout the world. Next planned work, will be to extend the data sizefor each subject, extend the number of subjects, and apply more advanced classifiers like deep learning classifiers that will include automatic feature selection and can exploit big data efficiently. The proposed finger writing recognition features, like pressure and writing style, can be aggregated and used to enhance existing biometric systems using techniques like in [49]. A future challenge, is to implement this FW system in real-time, connecting it directly to the watch or the cellphone in interactive manner, to be used as human-machine interface. REFERENCES [1] K. P. Feder and A. Majnemer, ‘‘Handwriting development, competency, and intervention,’’ Develop. Med. Child Neurol., vol. 49, no. 4, pp. 312–317, Apr. 2007. [2] J. W. P. Walker, D. T. G. Clinnick, and J. B. W. Pedersen, ‘‘Profiled hands in palaeolithic art: The first universally recognized symbol of the human form,’’ World Art, vol. 8, no. 1, pp. 1–19, Jan. 2018. [3] T. Schneider, G. Magyar, S. Barua, T. Ernst, N. Miller, S. Franklin, E. Montbach, D. J. Davis, A. Khan, and J. W. Doane, ‘‘P-171: A flexible touch-sensitive writing tablet,’’ in SID Symp. Dig., 2008, vol. 39, no. 1, p. 1840. [4] C. Gallea, S. G. Horovitz, M. ’Ali Najee-Ullah, and M. Hallett, ‘‘Impairment of a parieto-premotor network specialized for handwriting in writer’s cramp,’’ Hum. Brain Mapping, vol. 37, no. 12, pp. 4363–4375, Dec. 2016. [5] M. O’connor and D. S. Vannier, ‘‘Electronic pen device,’’ U.S. Patent 6 188 392, Feb. 13, 2001. [6] M. Nakai, T. Sudo, H. Shimodaira, and S. Sagayama, ‘‘Pen pressure features for writer-independent on-line handwriting recognition based on substroke HMM,’’ in Proc. 16th Int. Conf. Pattern Recognit., vol. 3, 2002, pp. 220–223. [7] B. Milner, ‘‘Handwriting recognition using acceleration-based motion detection,’’ IEE Colloq. Doc. Image Process. Multimedia, 1999, p. 5. [8] F. Zngf and S. Huang, ‘‘Glove virtual keyboard for baseless typing,’’ U.S. Patent Appl. 10/187 098, Jan. 1, 2004. [9] F. Kuester, M. Chen, M. E. Phair, and C. Mehring, ‘‘Towards keyboard independent touch typing in VR,’’ in Proc. ACM Symp. Virtual Reality Softw. Technol., 2005, pp. 86–95. [10] A. Butler, S. Izadi, and S. Hodges, ‘‘SideSight: Multi-touch interaction around small devices,’’ in Proc. 21st Annu. ACM Symp. User Interface Softw. Technol., 2008, pp. 201–204. [11] E. Whitmire, M. Jain, D. Jain, G. Nelson, R. Karkar, S. Patel, and M. Goel, ‘‘DigiTouch: Reconfigurable thumb-to-finger input and text entry on headmounted displays,’’ in Proc. ACM Interact., Mobile, Wearable Ubiquitous Technol., Sep. 2017, vol. 1, no. 3, pp. 1–21. [12] L. Chan, ‘‘FingerPad: Private and subtle interaction using fingertips,’’ in Proc. 26th Annu. ACM Symp. User Interface Softw. Technol., 2013, pp. 255–260. [13] S. Lee and S. Zhai, ‘‘The performance of touch screen soft buttons,’’ in Proc. SIGCHI Conf. Hum. Factors Comput. Syst., 2009, pp. 309–318. [14] B.-Y. Chen, ‘‘Finger-touch tracking system,’’ U.S. Patent 14 508 448, Apr. 7, 2016. [15] J. Liu, L. Zhong, J. Wickramasuriya, and V. Vasudevan, ‘‘uWave: Accelerometer-based personalized gesture recognition and its applications,’’ Pervasive Mobile Comput., vol. 5, no. 6, pp. 657–675, 2009. VOLUME 8, 2020 G. Blumrosen et al.: Back to Finger-Writing: Fingertip Writing Technology Based on Pressure Sensing [16] S. Nirjon, J. Gummeson, D. Gelb, and K.-H. Kim, ‘‘Typingring: A wearable ring platform for text input,’’ in Proc. 13th Annu. Int. Conf. Mobile Syst., Appl., Services, 2015, pp. 227–239. [17] R. Kaluri and C. H. Pradeep, ‘‘An enhanced framework for sign gesture recognition using hidden Markov model and adaptive histogram technique,’’ Int. J. Intell. Eng. Syst., vol. 10, pp. 11–19, 2017. [18] R. Kaluri and C. Pradeep Reddy, ‘‘A framework for sign gesture recognition using improved genetic algorithm and adaptive filter,’’ Cogent Eng., vol. 3, no. 1, Oct. 2016, Art. no. 1251730. [19] S. Heisig and K. Sakuma, ‘‘Characterizing primate nail deformation,’’ U.S. Patent 15 410 836, Mar. 29, 2018. [20] J. G. Da Silva, A. A. De Carvalho, and D. D. Da Silva, ‘‘A strain gauge tactile sensor for finger-mounted applications,’’ IEEE Trans. Instrum. Meas., vol. 51, no. 1, pp. 18–22, 2002. [21] S. A. Mascaro and H. H. Asada, ‘‘Photoplethysmograph fingernail sensors for measuring finger forces without haptic obstruction,’’ IEEE Trans. Robot. Autom., vol. 17, no. 5, pp. 698–708, Oct. 2001. [22] K. Sakuma, ‘‘Wearable nail deformation sensing for behavioral and biomechanical monitoring and human-computer interaction,’’ Sci. Rep., vol. 8, no. 1, 2018, Art. no. 18031. [23] K. Sakuma, G. Blumrosen, J. J. Rice, J. Rogers, and J. Knickerbocker, ‘‘Turning the finger into a writing tool,’’ in Proc. IEEE EMBS, Jul. 2019, pp. 1239–1242. [24] R. Williamson and B. J. Andrews, ‘‘Detecting absolute human knee angle and angular velocity using accelerometers and rate gyroscopes,’’ Med. Biol. Eng. Comput., vol. 39, no. 3, pp. 294–302, May 2001. [25] M. Koschinski, H.-J. Winkler, and M. Lang, ‘‘Segmentation and recognition of symbols within handwritten mathematical expressions,’’ in Proc. Int. Conf. Acoust., Speech, Signal Process. (ICASSP), vol. 4, 1995, pp. 2439–2442. [26] S. A. Gee, W. F. Van Den Bogert, and V. R. Akylas, ‘‘Strain-gauge mapping of die surface stresses,’’ IEEE Trans. Compon., Hybrids, Manuf. Technol., vol. CHMT-12, no. 4, pp. 587–593, Dec. 1989. [27] G. Blumrosen and A. Luttwak, ‘‘Human body parts tracking and kinematic features assessment based on RSSI and inertial sensor measurements,’’ Sensors, vol. 13, no. 9, pp. 11289–11313, 2013. [28] S. Ioffe and C. Szegedy, ‘‘Batch normalization: Accelerating deep network training by reducing internal covariate shift,’’ 2015, arXiv:1502.03167. [Online]. Available: http://arxiv.org/abs/1502.03167 [29] R. Plamondon and S. N. Srihari, ‘‘Online and off-line handwriting recognition: A comprehensive survey,’’ IEEE Trans. Pattern Anal. Mach. Intell., vol. 22, no. 1, pp. 63–84, Jan. 2000. [30] C. Pang, G.-Y. Lee, T.-I. Kim, S. M. Kim, H. N. Kim, S.-H. Ahn, and K.-Y. Suh, ‘‘A flexible and highly sensitive strain-gauge sensor using reversible interlocking of nanofibres,’’ Nature Mater., vol. 11, no. 9, pp. 795–801, Sep. 2012. [31] J. Han, L. Shao, D. Xu, and J. Shotton, ‘‘Enhanced computer vision with microsoft Kinect sensor: A review,’’ IEEE Trans. Cybern., vol. 43, no. 5, pp. 1318–1334, Oct. 2013. [32] J. Canny, ‘‘A computational approach to edge detection,’’ IEEE Trans. Pattern Anal. Mach. Intell., vol. PAMI-8, no. 6, pp. 679–698, Nov. 1986. [33] E. Amichai, G. Blumrosen, and Y. Yovel, ‘‘Calling louder and longer: How bats use biosonar under severe acoustic interference from other bats,’’ Proc. Roy. Soc. B, Biol. Sci., vol. 282, no. 1821, Dec. 2015, Art. no. 20152064. [34] V. Soto, O. Siohan, M. Elfeky, and P. Moreno, ‘‘Selection and combination of hypotheses for dialectal speech recognition,’’ in Proc. IEEE Int. Conf. Acoust. Speech Signal Process. (ICASSP), May 2016, pp. 5845–5849. [35] C. G. Goetz, ‘‘Movement disorder society-sponsored revision of the unified Parkinson’s disease rating scale (MDS-UPDRS): Process, format, and clinimetric testing plan,’’ Movement Disorders, vol. 22, no. 1, pp. 41–47, 2007. [36] Y. Lavi, D. Birnbaum, O. Shabaty, and G. Blumrosen, ‘‘Biometric system based on Kinect skeletal, facial and vocal features,’’ in Proc. Future Technol. Conf. (FTC), Vancouver, BC, Canada. Cham, Switzerland: Springer, Nov. 2018, pp. 884–903. [37] H. Sakoe and S. Chiba, ‘‘Dynamic programming algorithm optimization for spoken word recognition,’’ IEEE Trans. Acoust., vol. 26, no. 1, pp. 43–49, Feb. 1978. [38] H. Cornhill and J. Case-Smith, ‘‘Factors that relate to good and poor handwriting,’’ Amer. J. Occupational Therapy, vol. 50, no. 9, pp. 732–739, Oct. 1996. [39] G. Blumrosen and A. Freedman, ‘‘Sensitivity study and a practical algorithm for ML OSTBC and beam forming combination,’’ in Proc. 23rd IEEE Conv. Elect. Electron. Eng. Israel, Sep. 2004, pp. 1–5. VOLUME 8, 2020 [40] S. E. Levinson, ‘‘Continuously variable duration hidden Markov models for automatic speech recognition,’’ Comput. Speech Lang., vol. 1, no. 1, pp. 29–45, Mar. 1986. [41] W. Swaileh, Language Modelling for Handwriting Recognition. Caen, France: Normandie Univ., 2017. [42] J. Qi and Z. Yang, ‘‘Learning dictionaries of sparse codes of 3D movements of body joints for real-time human activity understanding,’’ PLoS ONE, vol. 9, no. 12, Dec. 2014, Art. no. e114147. [43] Y. Bassil and M. Alwani, ‘‘OCR post-processing error correction algorithm using Google online spelling suggestion,’’ 2012, arXiv:1204.0191. [Online]. Available: http://arxiv.org/abs/1204.0191 [44] K. S. Guillory and D. Yatsenko, ‘‘Self-contained surface physiological monitor with adhesive attachment,’’ U.S. Patent 11 827 387, Apr. 17, 2008. [45] G. Blumrosen, B. Hod, T. Anker, D. Dolev, and B. Rubinsky, ‘‘Enhancing RSSI-based tracking accuracy in wireless sensor networks,’’ ACM Trans. Sensor Netw., vol. 9, no. 3, pp. 1–28, May 2013. [46] A. Graves and J. Schmidhuber, ‘‘Offline handwriting recognition with multidimensional recurrent neural networks,’’ in Proc. Adv. Neural Inf. Process. Syst., 2009, pp. 545–552. [47] Y.-L. Hsu, C.-L. Chu, Y.-J. Tsai, and J.-S. Wang, ‘‘An inertial pen with dynamic time warping recognizer for handwriting and gesture recognition,’’ IEEE Sensors J., vol. 15, no. 1, pp. 154–163, Jul. 2015. [48] L. Prevost, C. Michel-Sendis, A. Moises, L. Oudot, and M. Milgram, ‘‘Combining model-based and discriminative classifiers: Application to handwritten character recognition,’’ in Proc. Proc. 7th Int. Conf. Document Anal. Recognit., 2003, pp. 31–35. [49] Y. Lavi, D. Birnbaum, O. Shabaty, and G. Blumrosen, ‘‘Biometric system based on Kinect skeletal, facial and vocal features,’’ in Proc. Future Technol. Conf., 2018, pp. 884–903. GADDI BLUMROSEN was born in Jerusalem, Israel. He received the B.S. and M.S. degrees in electrical engineering from Tel Aviv University, in 2005, and the Ph.D. degree in biomedical engineering from the School of Engineering and Computer Science, Hebrew University, in 2011. From 2012 to 2014, he held a postdoctoral position with the Computer Science Department, Tel Aviv University, in collaboration with the Zoology Department, and Tel Hashomer Hospital, producing new sensing and analysis tools in biology and medicine mainly to characterize human movement. In 2014, he was a Visiting Scholar with the Harvard Medical School, where he developed and implemented new tools to analyze the effect of electrical pulses for terminating cancer cells. In 2015, he was a Visiting Scholar with the Neuroscience Department, New York University, where he developed new tools to for non-primate facial recognition. In 2016, he joined the IBM Thomas J. Watson Research Center, to explore and quantize human gestures with new sensing and processing methods, in particular to detect scores to evaluate human and patient with Parkinson Disease performance. He is currently a Research Associate with the Faculty of Engineering, Bar Ilan University, where he works to improve methodologies in neural networks, inspired by the brain. 35467 G. Blumrosen et al.: Back to Finger-Writing: Fingertip Writing Technology Based on Pressure Sensing KATSUYUKI SAKUMA (Senior Member, IEEE) received the B.S. and M.S. degrees in mechanical engineering from Tohoku University, Japan, in 1998 and 2000, respectively, and the Ph.D. degree in nano science and nano engineering from Waseda University, Japan, in 2010. He joined IBM Research, Japan, in 2000. In 2011, he moved to the IBM Semiconductor Research and Development Center, East Fishkill, NY, USA. He transferred to the IBM Thomas J. Watson Research Center, NY, in 2013, where he is currently a Research Staff Member. He is currently a Visiting Professor with the Department of Biomedical Engineering, Tohoku University, Japan, and also with National Chiao Tung University, Taiwan. He has authored coauthored four book chapters, more than 85 peer-reviewed journal articles and conference proceeding papers, and more than 60 issued or pending U.S. and international patents. Dr. Sakuma has been serving as a Committee Member for the IEEE ECTC, since 2012, for the IEEE 3DIC, since 2016, and for the IEEE IRPS, since 2017. He was a recipient of the Exceptional Technical Achievement Award from the IEEE Electronics Packaging Society, in 2018, and the Alumni Achievement Award from the School of Engineering, Tohoku University, in 2017. He has been recognized with an Outstanding Technical Achievement Award (OTAA), in 2015, and the IBM Fourteenth Invention Achievement Award, in 2019. He is currently serving as an Associate Editor for the IEEE TRANSACTIONS ON CPMT. JOHN KNICKERBOCKER received the Ph.D. degree in materials science and engineering from the University of Illinois, in 1982. He has over 35 years’ experience at IBM in development and research. He is currently an IBM Distinguished Engineer, a member of IBM Academy, and a Master Inventor. He is also leading research on healthcare sensors and diagnostics in the Micro-System Technology and Solutions Team, IBM Thomas J. Watson Research Center, Yorktown Heights, NY, USA, with the goal of improved patient quality of life using precision healthcare monitors and diagnostics. He has authored or coauthored more than 300 patents and patent applications and has more than 85 technical articles, presentations, book chapters, and publications. Dr. Knickerbocker has received 75 technical awards from IBM and Industry for his work and inventions. He has over 30 years’ participation with industry technical societies across IEEE, ECTC, and ACerS. JOHN JEREMY RICE (Member, IEEE) received the B.S., M.S., and Ph.D. degrees in biomedical engineering from Johns Hopkins University, Baltimore, MD, USA, in 1987, 1989, and 1997, respectively. In 2001, he joined the IBM Thomas J. Watson Research Center, where he initiated the cardiac modeling program and championed neuroscience research efforts. He was a Senior Manager and a Principal Research Staff Member in IBM’s Healthcare & Life Sciences Organization, where he conducted research in cardiac modeling and multiscale modeling. In addition to his research, he was an Adjunct Assistant Professor with the Johns Hopkins School of Biomedical Engineering, contributed to The William J. Sacco Critical Thinking Foundation which mentors young people in STEM research. Recently, he led the Blue Sky Project, using real-time data of medical sensors like nail sensors, and accelerometers to monitor the progression of Parkinson’s disease in patients. He was an Active Member of the IBM Research Culture Club. 35468 VOLUME 8, 2020