JP6975952B2 - Living body motion identification system and living body motion identification method - Google Patents

Description

The present invention relates to a living body motion identification system for identifying a specific motion such as eating exercise performed by the living body and a living body motion identification method, and realizes a better life suitable for the living body, feedback to diagnosis / rehabilitation, and the like. The present invention relates to a living body motion identification system and a living body motion identification method useful for reporting an abnormal state to a living body.

Conventionally, as a motion identification system of this kind of living body, for example, the one published in Patent Document 1 (Japanese Unexamined Patent Publication No. 2003-220039) is known. This is a biometric information measuring device that measures at least one of the biometric information of the subject's blood pressure, pulse, respiratory rate, and blood oxygen concentration, and the subject's legs, arms, head, and chest. And an acceleration measuring device that measures the acceleration of multiple parts of the waist, the processor device calculates the exercise amount of the subject based on the acceleration information, and the threshold value of the biological information of the subject is calculated based on the exercise amount. The biological information of the subject is compared with the threshold value, and if the biological information exceeds the threshold value, the subject is notified of the occurrence of a physical abnormality. The processor device compares each of the biometric information of blood pressure, pulse, respiratory rate, and blood oxygen concentration with the calculated thresholds of blood pressure, pulse, respiratory rate, and blood oxygen concentration, and the plurality of biometric information is the same. If the threshold is exceeded, it is determined that the subject has a physical abnormality.
The threshold of biometric information is set in advance for each subject based on the calculated amount of exercise and the respective values of blood pressure, pulse, respiratory rate, and blood oxygen concentration at rest of the subject. There is. In addition, the processor uses a plurality of biometric information to indicate the movement and / or posture of the subject related to the warning (movement and / or posture in sleeping, sitting, sitting on a chair, walking, running, stair walking, bathing, eating, and excretion). ), And the necessity of abnormal treatment is judged based on the measured acceleration values of each of the multiple sites.

Japanese Unexamined Patent Publication No. 2003-220039

By the way, in the above-mentioned conventional system, the threshold value of the biological information is set for each of the calculated amount of exercise and the blood pressure, pulse, respiratory rate, and blood oxygen concentration in the individual subject, so that the setting is wide-ranging. There was a problem that it was complicated and the setting work was extremely complicated, and the movement and / or posture of the subject was judged from a plurality of biometric information, but the judgment was qualitative. It is difficult to detect fine movements, and therefore, for example, in the movement of meals, it is not possible to deal with fine movements such as signs that cause choking or aspiration.

The present invention has been made in view of the above points, so that the setting for identifying the operation can be easily performed without being complicated, and the detailed operation can be reliably handled. It is an object of the present invention to provide a living body motion identification system and a living body motion identification method.

The motion identification method for a living body of the present invention for achieving such an object is based on one or a plurality of motion detecting means for detecting a specific motion performed by the living body and a detection signal from the motion detecting means. In a biological motion identification system equipped with an identification processing unit that identifies a specific motion of
The identification processing unit is configured to include a machine learning means that discriminates a specific motion using all or part of the detection signal from the motion detection means and outputs the discrimination result.

Here, the specific motion includes a series of motions such as eating motion, sleeping motion, sitting motion, sitting motion, walking motion, running motion, bathing motion, excretion motion, etc., for example, facial expression-related motion, eyes. It is a concept that includes changes in parts of the living body such as movements of the body, breathing, body temperature, muscle movements, sounds such as voice, changes in the color of the skin, and internal movements that do not appear on the surface. It is also a concept that includes a stationary state.

Further, as the operation detecting means, for example, a photo sensor (photo interrupter (transmission type and actuator type), photo reflector (reflection type), etc.), a photoelectric element (photo register, LDR (light-dependent resistor)), a thermopile infrared sensor ( Thermoelectric power effect), pyroelectric infrared sensor, etc.), image sensor (linear image sensor, CCD, CMOS image sensor, etc.), optical remote control (light receiving module, etc.), optical sensor (photoelectron multiplying tube, image intensifier, etc.) ), Visual sensor (machine vision, color sensor, omnidirectional vision, binocular vision, active vision, etc.), olfactory sensor (smell sensor, odor code sensor, oil odor sensor, biosniffer, etc.), taste sensor (richness / sharpness sensor, etc.) ), Temperature sensor (body thermometer, semiconductor temperature sensor, platinum temperature sensor, thermista, thermocouple sensor, thermostat, etc.), temperature / humidity sensor (humidity sensor, temperature / humidity sensor, etc.), pressure sensor (blood pressure sensor, pressure sensor, tactile sensor, etc.) Piezoelectric elements (piezo), etc.), magnetic sensors (MR, GMR, TMR, MI, SQUID, etc.), human sensor (infrared, ultrasonic, visible light, etc.), proximity sensor, proximity switch, ultrasonic sensor, laser sensor, Fiber sensor, touch sensor, acceleration sensor (piezoelectric type, semiconductor, etc.), angular velocity sensor (tacogene, gyro sensor, etc.), vibration sensor, tilt / rotation sensor (gyro sensor, rotary encoder, resolver, servo tilt angle sensor, etc.), Position angle sensor (displacement sensor, length measurement sensor, linear scale, encoder, potency meter, etc.), azimuth angle sensor (attitude / orientation sensor, etc.), liquid level sensor (level sensor, etc.), liquid leakage / water detection sensor, flow rate sensor, Current sensors (myoelectric sensors, etc.), power sensors, biosensors (electrochemical biosensors, optical biosensors, etc.), face recognition sensors, safety sensors, motion sensors (optical, magnetic, inertial, etc.), counters (breathing) Camera devices such as electrocardiographs, electrocardiographic sensors, blood densitometers, microphones, depth sensors, pulse wave sensors, RGB sensors, and RGB-D camera devices can be mentioned. The camera device performs image processing on the image pickup data and outputs a detection signal related to the operation of the living body. From these, one or more are selected. The motion detecting means is not limited to the above.

According to this system, a specific motion is discriminated from the learning result by the machine learning means. Therefore, for example, even if there are a plurality of motion detecting means, a threshold value is not set for each of these motion detecting means as in the conventional case. However, each motion detection signal can be collectively related to a specific motion, so that the settings for identifying the motion can be easily performed without complicatedness, and fine motion can be reliably handled. become able to.

As a result, in the identification processing unit, for example, the machine learning means has learned in advance about a specific motion based on the detection signal from the motion detection means, and the learning result is that there is a detection signal from the motion detection means. A specific operation is discriminated from, and the discriminant result is output. Therefore, from this determination result, it can be confirmed that the specific operation has been performed in the case of the specific operation. You can also check other unplanned operations. In this case, if a specific operation is set to a normal operation, if it is determined that the specific operation has been performed, the normal operation is performed, and if it is another unplanned operation, it can be determined that the operation is abnormal, for example. Alternatively, for example, if normal operation and abnormal operation are set as specific operations, it is possible to confirm that either operation has been performed, and if the abnormal operation is performed, it can be determined that the operation is abnormal.

For example, as a specific movement of eating, in the sitting position, a hunched posture (decreased respiratory function), a cervical extension position (unnecessary tension in the shoulders and neck, narrowing of space to the esophagus), and a mouth. There are movements such as swelling (are you taking a bite?), Eating fast, stirring, rubbing, and drinking whole.
By setting these as specific movements and identifying them, isn't there a hunched posture (respiratory function decline) in the eating movement? Are you in the cervical extension position (unnecessary tension in the shoulders and neck, narrowing the space in the esophagus)? Do you maintain a personalized diet? (Example: Do you maintain sideways swallowing?), Are you taking an appropriate bite? , In what order are you eating solids and liquids? Do some patients perform alternate swallowing by alternately swallowing foods with different physical characteristics? Are you eating dangerously, such as eating fast, squeezing, rubbing, or drinking whole? , Is the meal pace good? Are you focused on your diet? (Are you vacant?) Are you talking while eating? , Is the breathing pattern changed during meals? And so on.

In the present invention, the machine learning means is an artificial neural network (ANN), a support vector machine (SVM), a decision tree, a random forest, k-means clustering, a self-organizing map, a genetic algorithm, and a Bayesian network. , It can be configured by one or a plurality of combinations selected from deep learning methods and the like.

Then, if necessary, the machine learning means creates teacher data related to a specific motion from all or part of the detection signals from the motion detection means in advance, and a judgment standard obtained by learning based on the teacher data. It is configured to have a learning function for storing data and an execution function for discriminating a specific motion corresponding to the stored determination criteria based on a detection signal from the motion detection means and outputting the discrimination result. Even if it is a comprehensive detection signal from the motion detecting means, it becomes possible to associate this with a specific motion on a one-to-one basis, so that a specific motion can be grasped easily and with high accuracy. Become.

In this case, the machine learning means is a specification obtained by labeling a feature amount extracting means for extracting a feature amount of a detection signal from the motion detecting means and a feature amount extracted by the feature amount extracting means at the time of learning. A judgment standard storage means for creating teacher data consisting of a label having a feature amount corresponding to the operation of the above and the corresponding feature amount and storing the judgment standard obtained by learning based on the teacher data, and the above-mentioned feature amount at the time of execution. It may be configured to include an estimation means for estimating a corresponding label from the feature amount extracted by the extraction means and the judgment criteria stored in the judgment criterion storage means, and an instruction means for indicating the label estimated by the estimation means. can.

According to this, for example, if a label corresponding to normal operation and a label corresponding to abnormal operation are provided by labeling, the corresponding label is estimated by the estimation means, and the corresponding label is indicated by the instruction means. From, it can be confirmed that either normal operation or abnormal operation has been performed.

Further, if necessary, the specific operation is configured to be specified by a single operation or a plurality of sets of a single operation. For example, in the case of a swallowing motion consisting of an opening motion, a chewing motion, a swallowing motion, and a closing motion, the opening motion, the chewing motion, the swallowing motion, and the closing motion are each specified as a single motion. Alternatively, a series of movements of opening movement → chewing movement → swallowing movement → closing movement can be specified as one movement.

Furthermore, if necessary, the specific operation is one or more of a normal operation performed by the living body, a dangerous operation that may cause an abnormality in the living body, and an abnormal operation different from the normal operation and the dangerous operation. It can be configured to be selected from.
By selecting dangerous movements that may cause abnormalities in the living body, it is possible to contribute to prevention of causing abnormal situations.

In addition, the normal operation and the dangerous operation can be configured to be a meal operation performed by the human body sitting in front of a table.
The eating movement can be configured to include a swallowing movement, a chewing movement, and an opening movement (feeding movement).

The above abnormal operation is configured to be a vital sign.
Vital signs include, for example, in the case of eating, swallowing upwards, facing downwards, falling down (suspicion of loss of consciousness due to suffocation, etc.), tapping the chest, and holding the neck with hands (universal choke sign). , Painful facial expression, facial color due to sudden thianose during nitrogen, swelling, coughing, wet cough (cough with sputum), dry cough, changes in SpO2 and breathing pattern, respiratory arrest, no asphyxiation (suffocation) Suspicion), tachycardia, descent and rise of heartbeat and pulse wave, moist cough, sputum-related voice, change to rattling voice (suspicion of aspiration), distressing grunt (suspicion of choking), etc. ..

Further, if necessary, the identification processing unit is provided with an alarm means for issuing an alarm when it is determined that the operation is at least an abnormal operation other than the normal operation.

Further, in order to achieve the above object, the present invention is a method for identifying a specific motion of a living body by using the motion identification system of the living body.

According to the present invention, since a specific motion is discriminated from the learning result by the machine learning means, for example, even if there are a plurality of motion detecting means, a threshold value is not set for each of these motion detecting means as in the conventional case. However, each motion detection signal can be collectively related to a specific motion, so that the settings for identifying the motion can be easily performed without complicatedness, and fine motion can be reliably handled. become able to.

It is a figure which shows the structure of the motion identification system of the living body which concerns on embodiment of this invention. It is a figure which shows the specific operation example which the operation identification system of the living body which concerns on embodiment of this invention identifies. It is a flowchart which shows the process at the time of learning of the machine learning means in the motion identification system of the living body which concerns on embodiment of this invention. It is a flowchart which shows the process at the time of execution of the machine learning means in the motion identification system of the living body which concerns on embodiment of this invention. It is a table diagram which shows the detector used for the detection item which concerns on 1st Embodiment of this invention. It is a figure which shows the system structure (the outline of the system structure of a multi-channel active electrode) which concerns on 1st Embodiment of this invention. It is a figure which shows the system configuration (the outline of the system which used the sensor) which concerns on 1st Embodiment of this invention. It is a figure which shows the system configuration (the outline of the system which used the magnetic unit) which concerns on 1st Embodiment of this invention. FIG. 3 is a diagram showing a state in which an accelerometer, an electromyogram, and a throat microphone are attached to a subject and the orientation of the acceleration sensor according to the first embodiment of the present invention. FIG. 3 is a diagram showing a state in which a multi-channel electrode is attached to a subject according to the first embodiment of the present invention. FIG. 3 is a diagram showing a state in which a magnetic sensor is attached to a subject according to the first embodiment of the present invention. FIG. 3 is a diagram showing a state in which the source of the magnetic unit is attached to a subject according to the first embodiment of the present invention. It is a table diagram which shows the specification of the sensor used which concerns on Example 1 of this invention. It is a figure which shows the outline of the identification method which concerns on Example 1 of this invention. It is a table diagram which shows the state of the frame shift which concerns on 1st Embodiment of this invention. It is a figure which shows the state of labeling according to 1st Example of this invention. It is a graph which shows the result of the preliminary experiment which concerns on 1st Example of this invention. It is a flowchart which shows the experimental operation which concerns on 1st Embodiment of this invention. It is a figure which shows the detection result of the event of the subject A which concerns on Example 1 of this invention. It is a figure which shows the detection result of the event of the subject B which concerns on Example 1 of this invention. It is a figure which shows the detection result of the event of the subject C which concerns on Example 1 of this invention. It is a figure which shows the follow-up result of the feeding pace which concerns on Example 1 of this invention. FIG. 3 is a diagram showing a state in which a sensor is attached to a subject according to a second embodiment of the present invention. It is a figure which shows the experimental result of Example 2 of this invention. It is a figure which shows the use example of this invention.

Hereinafter, the motion identification system for a living body and the motion identification method for a living body according to the embodiment of the present invention will be described in detail with reference to the accompanying drawings. Since the method for identifying the motion of a living body according to the embodiment of the present invention is realized by the motion identification system for the living body according to the embodiment of the present invention, it will be described in the description of this system.

As shown in FIG. 1, the motion identification system for a living body according to an embodiment of the present invention is one or a plurality of motion detecting means (T1, T2,) for detecting a specific motion performed by a living body (human body in the embodiment). T3 ... Tn) and an identification processing unit 1 for identifying a specific motion of a living body based on a detection signal from the motion detection means.

A particular action is specified by a single action or a set of single actions. Further, as shown in FIG. 2, the specific operation is one or more of a normal operation performed by the living body, a dangerous operation that may cause an abnormality in the living body, a normal operation, and an abnormal operation different from the dangerous operation. Is selected from. In the embodiment, any of normal operation, dangerous operation and abnormal operation is specified. For example, a specific movement is a meal movement performed by the human body sitting in front of a table, and normal and dangerous movements in this meal movement are normal meal movements performed by the human body sitting in front of a table. .. Eating movements include swallowing movements, chewing movements, and opening movements (feeding movements). The abnormal movement is a vital sign, and in the case of a meal movement, for example, a universal choke sign that holds the neck by hand is used.

A plurality of motion detecting means (T1, T2, T3 ... Tn) are provided in the embodiment, and examples thereof include a motion sensor, a microphone, a myoelectric sensor, an electrocardiographic sensor, a respiratory sensor, and a SpO2 sensor. Camera devices such as RGB-D camera devices that measure color images (RGB) and depth images (Depth) are also effective. The camera device performs image processing on the image pickup data and outputs a detection signal related to the operation of the living body.

The identification processing unit 1 discriminates a specific motion by using all or a part of the detection signals from the motion detection means (T1, T2, T3 ... Tn) and outputs the discrimination result. Machine learning means 2 It is configured with. In the embodiment, the machine learning means 2 is composed of a support vector machine (SVM). The machine learning means 2 creates teacher data related to a specific motion in advance from all or part (all in the embodiment) of the detection signals from the motion detection means (T1, T2, T3 ... Tn). Then, the learning function that stores the judgment criteria obtained by learning based on this teacher data and the judgment criteria stored based on the detection signals from the motion detection means (T1, T2, T3 ... Tn) correspond to the corresponding identification. It has an execution function that discriminates the operation of and outputs the discriminant result.

Specifically, the machine learning means 2 is extracted by the feature amount extracting means 3 for extracting the feature amount of the detection signal from the motion detecting means (T1, T2, T3 ... Tn) and the feature amount extracting means 3 at the time of learning. A teacher data consisting of a label having a feature amount corresponding to a specific motion obtained by labeling the obtained feature amount and the corresponding feature amount is created, and the judgment criteria obtained by learning based on the teacher data is stored. The estimation means 5 and the estimation means 5 that estimate the corresponding label from the feature amount extracted by the feature amount extraction means 3 at the time of execution and the judgment criteria stored in the judgment standard storage means 4 are estimated. It is configured to include an instruction means 6 indicating a label. For example, in the case of eating behavior, the labels classified into normal movement, dangerous movement, and abnormal movement are specified. The instruction means 6 indicates any of normal operation, dangerous operation, and abnormal operation.

The identification processing unit 1 is provided with a display means 8 for displaying the instruction result of the instruction means 6 on the display device 7. For example, in the case of a meal operation, the display of the display device 7 displays any one of normal operation, dangerous operation, and abnormal operation. Further, the identification processing unit 1 is provided with an alarm means 10 that displays a warning on the display device 7 and issues an alarm to the alarm device 9 when it is determined that the operation is dangerous or abnormal other than the normal operation. ..

Therefore, when the motion identification system of the living body according to the embodiment of the present invention is used to identify the motion of the living body, it is as follows.
<Learning>
As shown in FIG. 3, for example, in the case of a meal operation, the subject is made to perform a normal operation, a dangerous operation, and an abnormal operation, respectively, and the identification processing unit 1 collects standard data. The identification processing unit 1 acquires the detection signals in each case from the operation detecting means (T1, T2, T3 ... Tn) (S101), and extracts the feature amount by the feature amount extracting means 3 (S102). ). Then, this feature amount is labeled, a label having a feature amount corresponding to a specific operation (normal operation, dangerous operation, and abnormal operation) is specified (S103), and this label and the feature amount corresponding to this label are composed. Teacher data is created, and the judgment criteria obtained by learning based on the teacher data are stored in the judgment criteria storage means 4 (S104).

<Execution>
As shown in FIG. 4, when the identification processing unit 1 acquires the detection signal motion from the operation detection means (T1, T2, T3 ... Tn) (S201), the feature amount extraction means 3 features the detection signal. The amount is extracted (S202). The estimation means 5 estimates the corresponding label (normal operation, dangerous operation, and abnormal operation) from the feature amount extracted by the feature amount extraction means 3 and the judgment standard stored in the judgment standard storage means 4 (S203). The indicating means 6 indicates that the label is any of the corresponding normal operation, dangerous operation, and abnormal operation (S204). The display means 8 displays this (S205). Further, when the warning means determines that the operation is dangerous or abnormal other than the normal operation, the warning means displays a warning on the display device 7 and issues an alarm by the alarm device 9 (S205).

In this case, for example, a label corresponding to the normal operation, a label corresponding to the dangerous operation, and a label corresponding to the abnormal operation are provided by labeling, so that the label estimated by the estimation means 5 is indicated by the instruction means 6. Therefore, it can be confirmed that either normal operation, dangerous operation, or abnormal operation has been performed.

According to this system, the machine learning means 2 discriminates a specific motion from the learning result. Therefore, for example, even if there are a plurality of motion detecting means (T1, T2, T3 ... Tn), as in the conventional case. Even if a threshold value is not set for each of these motion detection means (T1, T2, T3 ... Tn), each motion detection signal can be collectively related to a specific motion, and therefore, the motion is identified. The settings for this can be easily performed without becoming complicated, and it will be possible to reliably handle detailed operations.

An example is shown below.
(1) First Example This embodiment is a system for identifying eating behavior, and is a system for detecting eating / swallowing, which is useful for preventing aspiration. Eating and swallowing are linked by various movements, and eating is roughly divided into the oral phase, pharyngeal phase, and esophageal phase. The oral period is the period of movement in which food is chewed in the oral cavity and mixed with saliva to form an easy-to-swallow mass (food mass) that is sent from the mouth to the pharynx. In the pharyngeal stage, the tip of the tongue is lifted, the hyoid bone reaches the pharynx, and then the hyoid bone is lifted.
It is the operating period until the bolus is guided to the esophagus. The esophageal period is the operating period from when the bolus is guided to the esophagus until the bolus is sent to the lower part of the esophagus. Since these are movements that occur in the oral cavity, it is difficult to identify the pharyngeal stage and the esophageal stage and to estimate the amount swallowed even if the oral stage can be visually confirmed. In addition, detection of openings is essential for identification of feeding pace. In this example, after redefining the oral phase as mastication and the pharyngeal phase and esophageal phase as swallowing, it was considered that feeding / swallowing consists of a cycle of opening and mastication, and swallowing. Furthermore, it was also necessary to estimate the feeding pace and the remaining amount of bolus in the oral cavity, and after taking these into consideration, the signals such as myoelectricity required for state detection were determined, and the system configuration was performed.

(1-1) Device determination Figure 5 shows the items to be detected, their usefulness, and the sensors used. In the oral, pharyngeal, and esophageal stages, the movement of the lower jaw, the muscle activity of the masticatory muscles and suprahyoid muscles, and the masticatory and swallowing sounds are expected to change, and information that contributes to the detection of specific movements is expected to be included. Therefore, an acceleration sensor, a myoelectric sensor, a multi-channel myoelectric sensor, and a throat microphone were attached. Although the data is not presented in this embodiment, an electrocardiograph, a SpO2 meter, an exhalation meter, and a magnetic sensor for detecting body movement were also used at the same time.

(1-2) System configuration The motion detection system used in this example is a triaxial accelerometer, an electromyogram for detecting masticatory muscle activity, and a multi-channel electrode for detecting suprahyoid muscle group activity. , Consists of a throat microphone. 6 to 8 show a schematic diagram of the system configuration. The sensors used for measurement include a three-axis accelerometer (ZB-150H, NIHON KOHDEN), an electromyogram (ZB-150, NIHON KOHDEN), and a multi-channel muscle for measuring surface myoelectric potential (EMG). An electromyogram and a throat microphone were used. In addition, although not used in this analysis, there are other ECG transmitters (ZB-151.H), respiratory transmitters (ZB-153H, NIHON KOHDEN), SpO2 transmitters (ZB-157H, NIHON KOHDEN), and Eight magnetic sensors were attached to the body to detect body movements, and data was collected at the same time. 9 to 12 show a state in which the sensor is attached. The accelerometer was mounted so that each axis faces in the direction shown in FIG.

The signal of the multi-channel electrode was measured by differentially amplifying the potential difference between the reference electrode attached to the earlobe and each electrode constituting the multi-channel active electrode with reference to the GND electrode attached to the other earlobe. ..
A data recording device (WEB-1000, NIHON KOHDEN) is used for triaxial acceleration and the surface myoelectric potential of the masticatory muscles, and an AD converter (NI USB-6218, NATIONAL INSTRUMENTS) is used for the surface myoelectric potential of the suprahyoid muscle group and the throat microphone. ) Was used to perform these synchronous measurements. FIG. 13 shows the specifications of various sensors.

(1-3) Meal movement identification method FIG. 14 shows a schematic diagram of a meal movement identification algorithm. The algorithm for motion identification is roughly divided into a "feature extraction unit for extracting features" and a "motor learning and identification unit by machine learning". In this embodiment, the feature quantity is extracted using the measured myoelectric signal, and the operation is identified by the support vector machine. Each part will be described in detail below.
(1-3-1) Feature extraction unit Before performing motion identification, characteristic signal components (features) related to motion are extracted from EMG [V] of the suprahyoid muscle group. For this feature, the root mean square (RMS), which is a feature of the time domain, and the Cepstrum coefficient (CC), which is a feature of the frequency domain, were used.
Root Mean Square (RMS)
The RMS is expressed by Eq. (1.), and the characteristics related to the amplitude of the EMG signal can be obtained.

Cepstrum coefficient (CC)
CC is expressed by equation (2).

It is a feature quantity extracted from the frequency domain, and has a feature that the envelope shape of the power spectrum and the fine structure can be separated. When the order is low, the characteristic of the envelope shape appears, and when the order is high, the feature of the fine structure appears.

The past n samples of EMG are used to calculate RMS and CC. At this time, a frameshift method is used in which n samples are cut out as one frame and calculated, and the cutout range is shifted at regular intervals. The state of performing a frame shift is shown in FIG. 15 below.

In this embodiment, the EMG signal was cut out while shifting the frame having a length of 128 ms (0.5 ms × 256) in a cycle of 16 ms (0.5 ms × 32), and the feature amount was extracted.
Furthermore, the moving average as shown in Eq. (3) was performed to smooth the features. Here, p is the frame number and M is the moving average score.

An action label is attached to the feature vector constructed using RMS and CC, and it is used for learning and identification.

(1-3-2) Labeling As shown in FIG. 16, it is necessary to perform labeling in learning by SVM. In this example, the movements of opening, chewing, and swallowing were identified. The labeling procedure and its appearance are shown below. Labeling was performed only by opening and swallowing, and chewing was performed between them. The upper waveform shows the EMG of the suprahyoid muscle group, the middle waveform shows the EMG of the masticatory muscle, and the lower waveform shows the sum of squares of each component of triaxial acceleration.
(1) Since it is necessary to open the mouth at the time of opening, the start of opening the first mouth is the part where the values of acceleration and muscle activity begin to change significantly.
(2) The end of the opening is the action of closing the mouth, and it is assumed that the values of acceleration and muscle activity have decreased, and the space between (1) and (2) is labeled as the opening.
(3) Since the start of swallowing is performed after the mastication is completed, the part where the activity of the masticatory muscles is significantly reduced is used.
(4) The end of swallowing was defined as the part where the activity of the suprahyoid muscle group following (3) was reduced, and swallowing and labeling were performed between (3) and (4).
(1) (2) (3) (4) are shown in order from the vertical line on the left side of FIG. Was shown as (c).

(1-3-3) Motor learning and discrimination unit First, it is necessary to construct a discrimination function using learning data. In SVM, there is a motion learning unit when used for learning and a motion identification unit that performs discrimination based on the learning result.
In the motion learning unit, the initial setting parameters of SVM are obtained from the learning data to which the motion class is given, and the discrimination function is constructed. γ and C are SVM hyperparameters, and the initial setting parameters γ and C are determined by grid search. The search range of γ and C is 48 combinations of γ = {,, ...,}, C = {,,, ...,}, and the combination showing the highest discrimination rate is searched from among the discrimination rates of each grid point. .. The discrimination rate at this time is obtained from the correctness between the discrimination result and the operation class of the data used for learning.
The motion identification unit identifies the feature vector based on the discrimination function created by learning, and assigns an motion class. After that, a majority decision is made for the past k given motion classes, and the state of eating and swallowing is finally determined.

(1-3-4) SVM multi-class extension
Since SVM is a method for identifying two classes, it is necessary to extend it to multiple classes when identifying many classes. Generally, there are two types of this method, the one-against-one method and the one-against-all method, but in this method, the one-against-one method was adopted. This is a method of constructing a combination of all O classes, that is, O (O-1) / 2 discriminant functions, and discriminating feature vectors using each discriminant function. The superiority of this method is shown by the comparative experiment on the learning time and discrimination accuracy of the two methods by Hsu et al.

(1-4) Identification of eating movements For young men, the EMG of the masticatory muscles during eating and swallowing, the acceleration due to jaw movement, the EMG of the suprahyoid muscles, and the swallowing sound were measured, and the above-mentioned eating movements were measured. The identification result is shown.
(1-4-1) Experimental conditions The subjects were 3 adult males with normal tongue function (age 23.0 ± 1.0 years, mean ± SD). As the above-mentioned operation for identifying the operation of eating / swallowing, the operation of eating / swallowing corn flakes, pudding, and water at normal times, the operation of eating / swallowing them at a reasonable speed, and the abnormal operation. As a sample of, a total of 5 actions were performed, entitled Cough 1 and clearing a cough close to a cough, and Cough 2 and a cough close to a corn. Each operation was performed 10 times as one set, and 2 sets were performed for each. The first set of each motion was used for learning SVM, and the second set was used for estimating each motion. This time, learning and identification were performed using the data of eating and swallowing corn flakes and the data of cough 1 and cough 2 as soon as possible and within a reasonable range.

A preliminary experiment was conducted in advance in which corn flakes were eaten and swallowed 10 times at normal speed. Preliminary experiments were performed with 6 subjects (age 22.5 ± 1.5 years, mean ± SD). Each of the 10 swallowing intervals was measured in the lap time format. The results are shown in FIG. The upper end of the result line for each subject is the longest mastication time, the lower end is the shortest mastication time, and the gray circle in the middle is the average value. As a result, since no common feeding / swallowing pace was found, it was judged that it differs depending on the person, and the experiment was conducted without specifying the operation cycle.
A flowchart showing a detailed experimental operation is shown in FIG. The measured and analyzed operations are shown in (*), and the measured and analyzed operations are shown in (**).

(1-4-2) Experimental result (1-4-2-21) Event detection result The event detection results of each subject are shown in FIGS. 19 to 21. Marks were placed on the areas where cough 2 was estimated, cough 1 was estimated, swallowing was estimated, mastication was estimated, opening was estimated, and no judgment was made. I put it in.

(1-4-2-2) Detection result of eating pace Below is the solid line when eating and swallowing at normal speed, and the dotted line and average value when eating and swallowing quickly within a reasonable range. The transition of is shown in FIG. 22 as a one-dot difference line.

(2) Second Example This embodiment is a system for identifying eating movements, and is an example relating to anomaly detection of eating movements.
The subjects were 5 healthy adult males (mean age 21 years, average height 171.0 cm). Physical movement was measured using a magnetic three-dimensional position / posture measurement system (LIBERTY, Polhemus Co.). This system consists of a control unit, a source, and multiple sensors, and by detecting changes in the magnetic field generated by the source, the position and orientation of each sensor with the source as the reference coordinate system is calculated. In this embodiment, as shown in FIG. 23, the source is fixed to the waist, and the sensor is fixed to the chest and the backs of both hands.

The measurement movements were general eating movements and abnormal movements. The subjects first simulated the movement of eating rice, drinking miso soup, drinking a drink in a glass, eating side dishes, and scraping rice, and finally the abnormal movement. I made one choke sign. Since the speed of meals varies from person to person, the time and number of meals were not specified. In addition, the chalk sign was performed while moving the upper body back and forth and left and right. These series of operations were regarded as one set, and a total of two sets were measured at a sampling frequency of 240 Hz.

The analysis method used SVM (Support Vector Machine), which is one of machine learning, to classify two classes of eating movement and abnormal movement from the measured three-dimensional position / posture data of each part of the body. The position coordinates of each sensor and the distance between the backs of both hands were used as the features. The motion label was given with 0 for meal motion and 1 for abnormal motion. The RBF kernel was adopted as the kernel function, and its hyperparameters ≡ and C were optimized by grid search. The search range of ≡ and C is 48 combinations of ≡ = {2-5, 2-4, ..., 20}, C = {21, 22, ..., 28}, and the identification rate of each grid point is selected. We searched for the combination that showed the highest discrimination rate. SVM was trained in the first set of measurements, and the discrimination accuracy was evaluated in the remaining one set. The identification result by SVM was smoothed by making a majority judgment of the past n samples.

In this example, the rate at which the choke sign was correctly identified as the choke sign was calculated as the accuracy. The calculation formula is shown below.

Identification rate
= (Correctly identified chalk sign and number of labels for meal movements) ÷ (total number of labels for all movements)
× 100 [%]

The identification results of the five subjects are shown in FIG. 24. The discrimination rate was distributed from 86.5 to 99.0%, and it was confirmed that eating behavior and abnormal behavior could be discriminated with an accuracy of 93.7 ± 4.1% on average.

In 2011, the third leading cause of death in Japan changed from "cerebrovascular disease" to "pneumonia". Of the 124,652 deaths from pneumonia, 96.5% are elderly people aged 65 and over, and about half of them are attributed to "aspiration pneumonia due to dysphagia." Dysphagia is a disorder in which a problem occurs in the process of swallowing due to the weakening of the muscles of the oral function, and food cannot be swallowed or food flows into the trachea (aspiration). In addition, it is said that about 30% of the late-stage elderly have dysphagia, and it is estimated that "the number of patients in Japan is at least 5 million." In addition, many elderly people are unaware of the decline in eating and swallowing disorders, and aspiration occurs by ingesting food that exceeds their abilities, such as choke signs that occur when suffocation or lumps of food become clogged in the throat. It is said that there are 5,000 cases per year in cases of abnormal operation and death due to airway closure (suffocation). Such accidents of aspiration and suffocation are increasing rapidly not only in the home but also in nursing care facilities such as nursing homes, and judicial precedents regarding nursing care accidents are becoming more common.

The present invention can be provided as a technique for supporting meal watching as a solution to this problem. Support for watching meals is to watch over meals so that people who are eating do not swallow, but there are problems such as a shortage of staff engaged in nursing care at nursing homes, and support for watching meals is a point to watch over. From the posture of the neck and trunk to the pace of eating, the amount of food eaten in one bite, and whether the patient's attention is distracted, it is extremely laborious to keep an eye on all of these. Conceivable. Actually, it is possible for an expert such as a caregiver to watch over by the side, or to monitor directly, but if a family member or a caregiver performs it concurrently with the usual work, fatigue will accumulate and the work will be done. It may cause problems. It is expected that the burden on caregivers and families will be reduced by automating meal monitoring support. In Japan, where problems such as the burden on caregivers have become apparent in the face of an aging society, it is expected that dysphagia and its surrounding problems will become more serious in the future, and urgent measures are indispensable. Is.

In the present invention, by detecting abnormal movements during meals and sensing movements that may lead to abnormal movements, it is possible to support meal watching as if a caregiver or other expert in meal support / guidance is there. You can aim to build an automatic system. For that purpose, it is necessary to automatically detect what is being eaten and what is being chewed in real time. From the results of automatic detection, it is possible to aim to build a system that gives guidance on how to eat as if an expert in food watching support is there, and sounds an alarm to family members and caregivers if they are behaving abnormally.

To prevent aspiration and suffocation in the elderly and people with dysphagia
"Eat food that is suitable for your swallowing ability at an appropriate pace and at an appropriate bite."
Is important,
"Dangerous movements such as sipping soup, stirring rice, and poor posture (cautionary movements)"
Has an extremely high risk of causing aspiration and airway obstruction. In aspiration during meals
"Museya cough"
However, in order to be aware of the deterioration of daily swallowing function and to perform effective rehabilitation,
"Record daily under what dietary conditions and how often aspiration occurs."
Is important. In the unlikely event that food blocks the airways and there are signs of suffocation, "promptly notify family members and caregivers."
is required. In order to monitor the diet of the elderly and dysphagia, it can be said that the state detection at each level shown in "" is indispensable, and the present invention can respond to this.

As shown in FIG. 25, the present invention uses SVM (support vector machine) to provide information on a motion sensor (3-axis acceleration, 3-axis angular velocity, 3-axis geomagnetism), myoelectric sensor, heart rate sensor, and microphone mounted on the wrist. By analyzing using machine learning methods (artificial intelligence) such as It can be provided as a technology for detecting each state such as a body sign.

T1 to Tn motion detection means 1 identification processing unit 2 machine learning means 3 feature quantity extraction means 4 judgment standard storage means 5 estimation means 6 instruction means 7 display device 8 display means 9 alarm 10 alarm means

Claims (10)

In a living body motion identification system including a plurality of types of motion detecting means for detecting a specific motion performed by a living body and an identification processing unit for identifying the specific motion of the living body based on a detection signal from the motion detecting means.
The specific operation is specified by an operation in which a plurality of sets of a single operation and the plurality of single operations are performed in a series.
The plurality of single motions are detected by one or more kinds of motion detecting means, and at least one single motion among the plurality of single motions is detected by two or more kinds of motion detecting means. death,
The identification processing unit is configured to include a machine learning means that discriminates a specific motion using all of the detection signals from the plurality of types of motion detection means and outputs the discrimination result.
The specific action is selected from one or two of a normal action performed by the living body and a dangerous action that may cause an abnormality in the living body.
The above-mentioned normal movements and dangerous movements are meal movements performed by the human body sitting in front of a table.
The above-mentioned meal movement is a movement identification system for a living body, which includes a series of movements of opening movement → chewing movement → swallowing movement → closing movement. The machine learning means of the identification processing unit is configured in advance with a function of creating teacher data related to a specific motion from all or part of the detection signals from the motion detection means.
In addition to the normal operation and the dangerous operation as the specific operation, an abnormal operation different from the normal operation and the dangerous operation is detected by one or more kinds of the operation detecting means.
The claim is characterized in that the machine learning means of the identification processing unit is configured to have a function of discriminating an abnormal operation by using all or a part of the detection signals from the motion detection means and outputting the discrimination result. Item 1. The biological motion identification system according to item 1. In a living body motion identification system including a plurality of types of motion detecting means for detecting a specific motion performed by a living body and an identification processing unit for identifying the specific motion of the living body based on a detection signal from the motion detecting means.
The specific operation is specified by an operation in which a plurality of sets of a single operation and the plurality of single operations are performed in a series.
The plurality of single motions are detected by one or more kinds of motion detecting means, and at least one single motion among the plurality of single motions is detected by two or more kinds of motion detecting means. death,
The identification processing unit is configured to include a machine learning means that discriminates a specific motion using all of the detection signals from the plurality of types of motion detection means and outputs the discrimination result.
The specific action is selected from one or two of a normal action performed by the living body and a dangerous action that may cause an abnormality in the living body.
The machine learning means of the identification processing unit is configured in advance with a function of creating teacher data related to a specific motion from all or part of the detection signals from the motion detection means.
In addition to the normal operation and the dangerous operation as the specific operation, an abnormal operation different from the normal operation and the dangerous operation is detected by one or more kinds of the operation detecting means.
The machine learning means of the identification processing unit is configured to have a function of discriminating abnormal operations using all or part of the detection signals from the motion detection means and outputting the discrimination results.
The above abnormal movements include swallowing upwards, face downwards, falling, tapping the chest, holding the neck with hands (universal choke sign), painful expression, facial color due to sudden coughing during nitrogen, and coughing. , Coughing, wet cough (cough with sputum), dry cough, changes in SpO2 and respiratory patterns, respiratory arrest, no elevation of the thorax, tachycardia, descent and rise of heartbeat and pulse waves, moist hoarseness, sputum entanglement A biological motion identification system characterized by a vital sign selected from a cough, a change to a rattling voice (suspicion of aspiration), and a distressing hoarseness. The motion identification system for a living body according to claim 3 , wherein the normal motion and the dangerous motion are meal motions performed by a human body sitting in front of a table. The movement identification system for a living body according to claim 4 , wherein the meal movement includes a series of movements of opening movement → chewing movement → swallowing movement → closing movement. The motion identification system for a living body according to any one of claims 2 to 5 , further comprising an alarm means for issuing an alarm when it is determined that the motion is at least an abnormal motion other than the normal motion. The above machine learning means are artificial neural network (ANN), support vector machine (SVM), decision tree, random forest, k-means clustering, self-organization map, genetic algorithm, Bayesian network, deep learning method. The motion identification system for a living body according to any one of claims 1 to 6, wherein the system is composed of one or a plurality of combinations selected from the above. The machine learning means has a learning function that creates teacher data related to a specific motion from all the detection signals from the motion detection means in advance and stores a judgment criterion obtained by learning based on the teacher data, and the motion. 6. Biological motion identification system. The machine learning means has a feature amount extracting means for extracting a feature amount of a detection signal from the motion detecting means and a specific motion obtained by labeling the feature amount extracted by the feature amount extracting means at the time of learning. A judgment standard storage means that creates teacher data consisting of a label having a corresponding feature amount and the corresponding feature amount and stores the judgment criteria obtained by learning based on the teacher data, and the above-mentioned feature amount extraction means at the time of execution. It is characterized in that it is configured to include an estimation means for estimating a corresponding label from the extracted feature amount and the judgment criteria stored in the judgment criterion storage means, and an instruction means for indicating the label estimated by the estimation means. The motion identification system for a living body according to claim 8. A method for identifying an action of a living body, which comprises using the motion identification system of the living body according to any one of claims 1 to 9 to identify a specific action of the living body.

Images