Roughness Encoding in Human and Biomimetic Artificial Touch: Spatiotemporal Frequency Modulation and Structural Anisotropy of Fingerprints

2.1. Biomimetic Fingertip

Four MEMS force micro-sensors [22,23] were integrated in a 2 × 2 array via flip-chip bonding on a rigid-flex board. Each sensor of the array was bonded on the rigid part on the corner of a square with a pitch ΔX of 2.36 mm, allowing the four tethers of each sensor to be suspended and free to flex under externally applied loads while the rigid support guaranteed stable mechanical bonding. Each sensor integrated four piezoresistors as sensing elements at the roots of the tethers forming a cross shape structure equipped with a mesa. This resulted in an array with 16 channels in total for transducing the mechanical interaction with external tactile stimuli. Inscribing each sensor in a square of area 5.57 mm², a 0.72 channels/mm² (16 channels/22.28 mm²) density was achieved, which mimics the SAI innervation density in humans (70 units/cm²) [7].

The 16 channels of the array were acquired by means of a high resolution (24 bit) Analog to Digital Converter (ADS1258, Texas Instruments, USA). The integration of the ADC onboard the fingertip allowed to reduce the amount of wires between the fingertip and the outer electronics, requiring power supply and a few digital communication channels only, and also guaranteed adequate signal-to-noise ratio due to the limited length of the connections routing analog signals. Data was sampled at 250 Hz per channel since such value was about one order of magnitude higher than the expected fundamental frequencies (as from Equation (1)); however, higher sampling frequencies are allowed by (1) increasing the overall conversion rate (this operation will affect the signal to noise ratio, but the achieved S/N levels guarantee that this is feasible) of the ADC lodged onto the fingertip, or (2) by reducing the number of converted channels (this operation will not affect S/N) without changing the overall conversion rate of the ADC. Acquired data was transmitted to a PC via Ethernet protocol by a soft-core processor (NiosII, Altera, USA) instantiated onboard a FPGA (Cyclone II, Altera, USA).

The rigid-flex board with MEMS sensor array and readout electronics was integrated in a rigid fingertip mimicking human anthropometry (Figure 1). The fingertip was designed for application to distal phalanxes of robotic hands being appropriate for grasping and manipulation tasks in anthropomorphic manner [24,25] and was fabricated with rapid prototyping resin via a 3D printer.

The packaging skin-like layer of the 2 × 2 array of MEMS sensors was introduced to have a similar function to the epidermal ridges of a human finger that can enhance deformation and frictional properties of the fingertip surface. Significant contributions in the simulative analysis [26,27] and artificial emulation [27,28] of fingerprints were given by Maeno and colleagues, showing that their structure increases the sensitivity in tactile activities with a major effect on surface located type I receptors. Therefore, fingerprints were included in the design of the proposed biomimetic fingertip considering that the epidermal ridges and grooves of an adult human have width in the 100–300 μm range, and the typical between-ridge distance is 400 μm [8,26,28].

The encapsulation was performed by means of soft polymeric packaging (Dragon Skin, Smooth-On, USA), having shore A 10 hardness and recovering its original form after a mechanical stimulation. The packaging material was poured directly on the fingertip by means of a mould that allowed to pattern the surface of the skin-like layer on top the sensor array.

Each single sensor of the array provides local information on the contact interaction at its interface with the surrounding polymeric packaging material, with the advantages of distributed tactile sensing [29]; in addition, our array of tactile sensors provides also directional information by means of the output readings from the four piezoresistors (implanted each at a root of a tether). Therefore, to have four outputs from each sensor increases the informative content on the stress state locally at each sensor site.

In order to investigate the role of the shape of fingerprints in texture encoding, two curvatures were designed (Figure 1(A,B)). In finger a, fingerprints were embossed with straight parallel ridges having between-ridge distance Δp_f set to 400 μm. Finger b had concentric fingerprints with groove and ridge widths as for prototype a, and the fingerprint passing from the center of the sensor array had curvature radius of 4.8 mm.

As regards the thickness, here we used an artificial epidermal ridge with a height h₁ of 170 μm, while the thickness h₂ of the homogeneous packaging layer covering the sensor array was 600 μm (Figure 2); this resulted in sensing units being located quite close to the surface of the fingerpad, similarly to the positioning of type I human mechanoreceptors [26,28].

Preliminary load-unload tests with smooth flat surfaces were performed (not shown in this work) and showed that, for both the fingertip designs, the design of the packaging allows to reach at least up to 8 N normal and 4 N tangential forces without any damage to the encapsulated sensors.

2.2. Microneurography Technique for Human Touch Studies

Impulses of single tactile afferents in the left index and middle fingers were recorded using the microneurographic technique in 36 human healthy volunteers [2]. The subjects seated comfortably in a dentist’s chair, the left arm resting in a vacuum cast for stabilization and maximum comfort. Tungsten needle electrodes were inserted in the left median nerve, 8 cm above the elbow. The nerve signal was band-pass filtered at 200–4,000 Hz, sampled at 12.8 kHz together with analog data from the tactile stimulation mechatronic platform, and stored on a PC using the ZOOM/SC system developed at the Department of Physiology, Umeå University, Sweden. Recorded nerve impulses were inspected off-line on an expanded time scale using in-house software implemented in MATLAB (The Mathworks) and were accepted for subsequent analyses only if they could be validated as originating from a single afferent. Before running the experimental protocol, the units’ responses and receptive fields were explored using calibrated nylon filaments (von Frey hairs) and were classified as SAI, SAII, RA, or PC according to the adaptation of the response to sustained stimulation and size of the receptive field [30,31].

2.3. Experimental Set-Up and Protocol

The study focused on experimenting passive-touch protocols in which periodic ridged stimuli were indented (z direction, Figure 1(C)) and slided (y direction) on the human fingertip and on the two artificial fingertip prototypes (differing in the curvature of fingerprints).

The passive-touch stimulation sequences were implemented by means of a mechatronic platform [32] with which repeatable experiments could be performed (Figures 1(C) and 3). This consisted in a 2 DoF system that could indent and slide textured stimuli to the fingertip. The system performed a feedback control on the normal contact force (i.e., indentation along the z direction) and a precise position/velocity control while recording the normal and tangential forces at finger-stimulus interface. The tactile stimulator could present stimuli to the fingertip without being affected by spurious vibrations and covering a range of forces and movement velocities as those that would be used by humans while exploring textures [4,6], i.e., at least from 100 mN up to 5 N indentation force and up to 150 mm/s tangential sliding velocity. The same core mechatronic tactile stimulator was used for human (Figure 3(A)) and artificial touch (Figure 3(B)) experiments. Periodic ridged surfaces (gratings), fabricated from tufset rigid polyurethane thermosetting plastics material, were tested as tactile stimuli in both human and artificial touch experiments.

3.1. Human Touch Data Analysis

Neural data was at first inspected and processed in time domain to identify neural events. Then, the identified spikes were analyzed by obtaining spectra of the spike trains from single afferent units using the approach for frequency domain analysis for point processes [37]. For frequency domain analysis, 1.0 second data windows without overlap were used, and spectra were obtained from disjoint windows for all the available data for a specific stimulation condition (i.e., combination of sliding velocity, surface type, and normal force). Quantitative analysis was then performed across all the acquired data to evaluate whether or not a specific class of tactile units presented a spectrum with a principal frequency according to Equation (1).

3.2. Artificial Touch Data Analysis

After a preliminary graphical inspection of data from the biomimetic fingertip in time domain, the core analysis was operated in time-frequency domain via Short Time Fourier Transform (STFT) and in frequency domain via Fast Fourier Transform (FFT) over 1,024 samples (4.096 s of data at 250 Hz) in the middle of the stimulus sliding phase.

A recent study in active touch showed outstanding repeatability of experimental data by means of the fingertip having curved fingerprints [36]. Those previous results allowed us to directly consider here the aggregated results rather than data on a single run basis, also because real-time is not targeted in this work. Therefore, the six repeated runs per combination were aligned off-line and averaged channel by channel, enhancing their statistical significance and emphasizing their informative content.

For quantitative analysis of biomimetic fingertip experimental results, the principal spectral component f_peak is retrieved from outputs of the sensor array in order to evaluate the matching with the expected frequency f_princ (Equation (1)):

4.1. Human Touch

Figure 5 shows sample nerve recordings, gathered from the median nerve above the elbow using the microneurographic technique [2], from a human RA receptor during stimulus sliding motion across the distal phalanx (i.e., θ = 90°). The subject’s fingerprint at the location of the depicted RA unit has a tangent oriented at approximately 46° from the direction parallel to the ridges of the gratings (=44° degrees from the direction of the sliding motion). Spectral analysis of the nerve discharge patterns [37] showed significant modulation at the frequency determined by the stimulus spatial period Δp_s, according to Equation (1). A similar relationship depending on the stimulus spatial period was observed in the activity of single human mechanoreceptors with receptive fields in the finger tips of the second and third fingers. Particularly, for the tested gratings in the 280–520 μm spatial period range, this frequency-locked modulation was for 8 of 9 RA afferents units where this was tested, but not in any of the SAI units (n = 5). For gratings in the 1,600–1,920 μm spatial period range, it was observed in all of the tested RA and SAI units (n = 7 and 5, respectively; 10–20 mm/s sliding velocity). The smaller peak at 30.8 Hz in Figure 5(C) reflects a slight periodic modulation of unit firing that is uncorrelated with the periodicities in the mechanical stimulus. Moreover, it should be noted that the average discharge rates of single tactile afferents never directly reflected the spatial periods of the stimuli. As an example, average discharge rate was 40 Hz for the unit in Figure 5(A), and 55.5 Hz for the unit in Figure 5(B). Thus, there was no 1:1 (or higher order) locking of the nerve discharges, but the spatial periodicity were reflected as a frequency modulation (Equation (1)) of the discharge patterns.

4.2. Artificial Touch

Figure 6 shows the output signal from a channel of the tactile array over six runs recorded under the same experimental conditions, in order to assess high repeatability of data. Cursory analysis of Figure 6 confirms high similarity among the plots. As a further quantitative assessment, Table 1 presents the correlation indexes calculated over all the pairs of the runs plotted in Figure 6. All the correlation values reported in Table 1 are close to 1, confirming high repeatability. Similar repeatable results were obtained from the other channels of the array, for both the fingerprint designs and for all the experimental combinations (θ and Δp_s values). High repeatability allowed to perform the averaging operation for all the following experimental results, as detailed in Section 3.2, in order to provide significant information by means of data recorded under multiple runs.

In Figures 7 and 8 time domain data from single channels (Piezoresistor 2 of Sensor 1 and Piezoresistor 4 of Sensor 4, respectively) of the experimented biomimetic fingertip designs is plotted above the related STFT. The insets on the right of the STFT plots show the spectra obtained by applying a FFT to the single channel data highlighted in red in the time domain plots.

Particularly, Figure 7(A,B) shows time domain traces from Piezoresistor 2 of Sensor 1 in fingertip b (see Figure 1 for the labeling of sensors of the tactile array) during stimulation with 360 μm and 440 μm regular gratings rotated at an angle θ = 10°. The periodic patterns at 27.8 Hz (360 μm grating) and at 22.7 Hz (440 μm grating) associated to the spatial periodicity of tactile stimuli are clearly visible either in time (vibrational component), in frequency (dominant peak in the FFT, marked with a dotted line) and in time-frequency (red region, marked with a dotted line in the STFT) domains. Since the sliding velocity remains constant in the performed experiments, the dominant frequency of the vibrations elicited by the tactile stimulus is proportional to the inverse of the spatial period of the grating (Equation (1)), while the intensity of the vibrations increases with the spatial period. Both these effects appear to be coherent with [13], where the mechanical vibrations recorded in the fingertip of human subjects are shown to scale down in peak frequency and to increase in peak-to-peak amplitude while increasing the spatial period.

The relevance of the dynamic stimulation phase (i.e., the dataset corresponding to surface-fingertip tangential relative motion) to extract vibrational patterns which are correlated to the stimulus surface features is confirmed by the STFT spectrograms depicted below the time domain plots, which show a sudden frequency step at the onset of the stimulus sliding motion. The spectral pattern remains stable while the periodic grating is stroked at constant velocity. More importantly, as confirmed by the FFT spectra, the frequency peak corresponds to the expected value depending on the applied stimulus according to Equation (1), i.e., 27.8 Hz for the 360 μm surface and 22.7 Hz for the 440 μm one. Significantly, this artificial vibrational roughness encoding is coherent with the microneurography results in humans, according to the findings reported in Section 3.1 and to previous studies with monkeys [38].

A comparison between the A–D and the E–H panels in Figure 8 shows the effect of the curvature of fingerprints in the encoding of stimulus spatial features with respect to the rotation of the biomimetic fingertip. The four rows show results for θ = 10°, θ = 20°, θ = 40° and θ = 90°. There is higher isotropy with the curved fingerprints than with the straight ones, which have a strongly preferred direction when the sliding is closer to the direction along the distal phalanx (i.e., across the fingerprints). As shown in Figure 8(A), with straight fingerprints the vibratory patterns are noticeable either in time and frequency domains for θ = 10°, while those patterns are considerably reduced and masked by the other spectral components when the fingertip is rotated (Figure 8(B–D)) so to have a sliding oriented closer to the direction across the distal phalanx (i.e., along the fingerprints).

Extended analysis of the spectrum of readings from both the biomimetic finger designs as a function of the rotation angle θ brings evidence of the higher anisotropy anticipated above for straight fingerprints (Figure 9(A) compared to Figure 9(B) and Figure 10(A) compared to Figure 10(B)). Within the plots shown in Figures 9 and 10, the expected (Equation (1)) principal frequency is represented by a straight red line, while the peak arising in the frequency domain for output PiSj (i^th piezoresistor of j^th sensor, according to Figure 1) is detected by applying Equation (2) and the correctly identified ones are marked with red circles in the figures.

It is significant to point out that, differing from Figure 9(A) (straight fingerprints), in Figure 9(B) (curved fingerprints) the peak is not at θ = 0° but at θ = 10°. This is a consequence of the curvature of fingerprints, which affects the sensitivity of the packaged system in tradeoff with the preferred direction of Piezoresistor P4, hence widening the region of effective roughness encoding as a function of the finger rotation angle θ: the tangent to the curved fingerprints at the location of Sensor S4 is orthogonal to the x axis sliding motion direction (and contemporarily parallel to ridges of the grating) when θ = 17.4° (i.e., >10°), while piezoresistor P4 shows its maximum sensitivity [23,39] to tangential loads (reaching the sensor through the packaging material) being oriented along the direction of its tether, which is aligned with the x axis when θ = 0° (i.e., <10°).

Considerations similar to those reported above for Figure 9 apply to Figure 10 as well, which depicts spectral data from P2S1 as a function of the rotation angle θ. The tangent to the curved fingerprints at the location of Sensor S1 is orthogonal to the x axis sliding motion direction (and contemporarily parallel to ridges of the grating) when θ = 10.9°, which is lower than the related value for S4; as a consequence, in Figure 10(B) the peak at θ = 10° appear to be more marked than the one at θ = 20°, if compared with the same pair of peaks in Figure 9(B).

For all the channels of the tactile array, the design with straight fingerprints guaranteed an absolute error Δf = |f_princ − f_peak| in principal frequency estimation (via Equation (2)) lower than 0.15 Hz for 75% of all the experimental combinations, while the percentage raised to 82.5% for the design with curved fingerprints.