CN114466618A - Stress coping style determination system and method, learning device and method, program and learning completion model - Google Patents

Abstract

The invention provides a system capable of judging a pressure response mode of a subject in a non-contact state. Comprises the following components: a biological information acquisition unit that acquires biological information of a subject in a non-contact state; and a determination unit configured to determine a stress response mode of the subject based on the biological information and a predetermined response pattern. The response mode is determined by hemodynamic parameters.

Description

Stress coping style determination system and method, learning device and method, program and learning completion model

technical field

The present invention relates to a technique for judging a stress response form of a subject in a non-contact state.

Background technique

The technique of grasping the stress state of a subject is a well-known technique. For example, Patent Document 1 describes a technique of estimating the psychological change of a subject by measuring the thermal radiation heat of the face.

In addition, Patent Document 2 describes a technique for performing level measurement of the psychological state of a subject based on facial image information. According to these techniques, it is possible to detect a certain psychological change in the subject in a non-contact state, or to quantitatively analyze the psychological state of the subject in a non-contact state.

However, in Patent Documents 1 and 2, the type of stress felt by the subject cannot be grasped, and therefore the nature of the stress of the subject cannot be analyzed.

prior art literature

Patent Literature

Patent Document 1: Japanese Patent Application Laid-Open No. 6-54836

Patent Document 2: Japanese Patent Laid-Open No. 2007-68620

SUMMARY OF THE INVENTION

The technical problem to be solved by the invention

Humans are known to exhibit characteristic response patterns in order to achieve the goal that the cardiovascular system meets the metabolic demands from various tissues of the body when faced with stressful stimuli. That is, a mode showing active coping, a mode showing passive coping, and a mode showing no specific coping with stress. These modes are referred to as "stress coping modes", and when a mode of active coping occurs, it can be presumed that the subject is in a good stress state. Conversely, when the passive coping mode appears, it can be presumed that the subject is in a poor stress state.

That is, by determining the stress coping style of the subject, the type of stress felt by the subject can be grasped.

The present invention provides a stress coping mode determination system, a stress coping mode determination method, a learning apparatus, a learning method, a program for realizing these using a computer, and a learned model capable of determining the stress coping mode of a subject in a non-contact state.

Solutions for solving the above technical problems

In order to solve the above-mentioned technical problem, the stress coping method determination system of claim 1 is characterized by comprising: a biometric information acquisition unit that acquires biometric information of the subject in a non-contact state; and a determination unit that is based on the biometric information and a predetermined The determined response pattern, which is determined by hemodynamic parameters, determines how the subject copes with stress.

The stress coping mode determination system acquires the subject's biological information in a non-contact state, and determines the subject's stress coping mode in a non-contact state based on the biological information and the response mode determined by the hemodynamic parameters.

In addition, the stress coping mode determination system of claim 2 is characterized in that, in the stress coping mode determination system of claim 1, the hemodynamic parameters include mean blood pressure, heart rate, cardiac output, stroke volume, and total Multiple parameters in peripheral vascular resistance.

Here, "hemodynamics" refers to a branch of circulatory physiology that takes blood circulation as the object, and is a field of research that applies the theories of mechanics, elastic dynamics, and fluid dynamics to biological systems.

Specifically, cardiac internal pressure, pulsation, work capacity, stroke volume, elasticity of blood vessels and myocardium, pulse, blood flow velocity, blood viscosity, and the like are studied. Therefore, in the present invention, "hemodynamic parameters" refer to parameters such as the internal pressure of the heart, beating, work capacity, stroke volume, elasticity of blood vessels and myocardium, pulse, blood flow velocity, blood viscosity, etc. variable.

The stress response mode determination system acquires the biological information of the subject in a non-contact state, and compares the biological information with a plurality of parameters including average blood pressure, heart rate, cardiac output, stroke volume and total peripheral vascular resistance The determined response mode is used to determine the stress coping style of the subject. Furthermore, hemodynamic parameters can often be identified by continuous sphygmomanometers.

Furthermore, the stress coping mode determination system of claim 3 is characterized in that, in the stress coping mode determination system of claim 2, the biological information is a face image.

The stress coping mode determination system acquires a facial image of the subject in a non-contact state, and determines the stress coping mode of the subject based on the facial image and the response mode.

In addition, the stress coping mode determination system of claim 4 is characterized in that, in the stress coping mode determination system of claim 3, the facial image is a thermal facial image or a visible facial image.

The stress coping mode determination system acquires the subject's thermal facial image or face visible image in a non-contact state, and determines the subject's stress coping mode based on the facial thermal image or visible face image and the above-mentioned response mode.

In this case, the "face visible image" is an image obtained by photographing the face of the subject with a generally widely used camera, that is, a device having an optical system for imaging and photographing an image. In this case, a color image is preferable. In addition, the "facial thermal image" is an image which analyzes infrared rays radiated from the face of the subject, and displays the thermal distribution as a graph, and is an image captured by an infrared thermal imager.

Furthermore, the stress coping mode determination system of claim 5 is characterized in that, in the stress coping mode determination system of claim 3 or 4, the determination unit observes the stress response of a specific part of the face included in the face image, to determine the subject's coping style of stress.

The stress coping mode determination system determines the stress coping mode of the subject based on the stress response of a specific part of the subject's face and the response pattern.

Furthermore, the stress coping mode determination system of claim 6 is characterized in that the response mode includes three modes including "active coping", "passive coping", and "no coping".

The stress coping mode determination system acquires the biological information of the subject in a non-contact state, and based on the biological information, determines whether the stress coping mode of the subject shows "active coping", "passive coping" and "no coping" " in which way to respond to the pattern.

In addition, the stress coping mode determination system of claim 7 is characterized in that, in the stress coping mode determination system of claim 6, the determination unit has a feature of storing a spatial feature value corresponding to "active coping" and corresponding to "passive coping". The feature value storage unit for determination of the spatial feature value and the spatial feature value corresponding to "no response", based on the biometric information and each spatial feature value stored in the The stress coping mode is a mode indicating which response mode is "active coping", "passive coping", and "no coping".

The stress coping mode determination system stores the spatial feature quantity corresponding to "active coping", the spatial characteristic quantity corresponding to "passive coping", and the spatial characteristic quantity corresponding to "no coping", based on the biological information of the subject and each The spatial feature value is used to determine which response mode of "active coping", "passive coping", and "no treatment" is indicated by the stress coping mode of the subject.

In addition, the stress coping mode determination system of claim 8 is characterized in that, in the stress coping mode determination system of claim 7, the spatial feature quantity stored in the determination feature quantity storage unit is a space extracted by a machine learning unit feature quantity, the machine learning part has: a learning data storage part, which stores a plurality of face images for learning with labels corresponding to "active response", "passive response" and "no response" respectively; a feature value extraction part, Use the learning completed model to extract the spatial feature of the face image from the face image for learning; the feature learning part is based on the extraction result obtained by the feature extraction part and the learning object as its extraction object. The relationship between the labels attached to the facial image, and the network parameters of the learned model are changed so that the extraction accuracy of the spatial feature obtained by the feature extraction unit is increased.

In this stress coping mode determination system, the machine learning unit extracts a spatial feature amount corresponding to "active coping", a spatial feature amount corresponding to "passive coping", and a spatial feature amount corresponding to "no coping".

The machine learning department stores a plurality of face images for learning with labels corresponding to "active response", "passive response" and "no response", and uses the learned model to extract the face image of the subject from the face image for learning. Based on the relationship between the extraction result and the label attached to the learning facial image to be extracted, the network parameters of the learned model are changed so that the extraction accuracy of the spatial feature of the subject's facial image is improved.

In addition, the stress coping mode determination system of claim 9 is characterized in that, in the stress coping mode determination system of claim 7 or 8, the spatial feature amount is a fractal dimension calculated based on a face image of the subject.

In addition, the program of claim 10 is a program for causing a computer to function as a means for determining a stress coping style of a subject, and is characterized by comprising a step of storing the feature value for determination, and storing a space corresponding to "active coping" feature amount, spatial feature amount corresponding to "passive response", and spatial feature amount corresponding to "no response"; determination step, based on the face image of the subject and each spatial feature stored in the feature amount storage step for determination It is determined which response mode of "active coping", "passive coping" and "no coping" is shown in the stress coping mode of the subject, which is determined by the hemodynamic parameters.

The program is installed and executed in one or a plurality of computers working in cooperation with each other, so that a system composed of the one or a plurality of computers working in cooperation with each other functions as a mechanism based on the spatial feature amount corresponding to "active response" , the spatial feature quantity corresponding to "passive coping", the spatial feature quantity corresponding to "no coping", and the face image of the subject, and determine whether the subject's stress coping style shows "active coping", "passive coping" And what kind of response mode in "no response".

In addition, the program of claim 11 is characterized in that, in the program described in claim 10, the program includes a step of storing data for learning, and storing a plurality of labels corresponding to "active response", "passive response", and "no response", respectively. Facial image for learning; Feature extraction step, using the learning completed model to extract the spatial feature of the facial image for learning; Learning step, based on the extraction result obtained by the feature extraction step and the extraction object. The learning of the relationship between the labels marked with the facial image, the network parameters of the learned model are changed so that the extraction accuracy of the feature obtained by the feature extraction step becomes high, and the determination feature storage step is the step of storing the spatial feature quantity extracted by the feature quantity extraction step.

Install and execute the program in one or a plurality of computers working in cooperation with each other, so that the system composed of the one or more computers functions as the following system: storage and "active response", "passive response" and "no response" "respectively correspond to multiple face images for learning with labels, use the learned model to extract the spatial feature of the facial images for learning, and change the learning based on the relationship between the extraction results and the labels labeled with the facial images for learning that are the extraction objects. The network parameters of the model are completed so that the extraction accuracy of the spatial feature amount becomes high, and the extracted spatial feature amount is stored.

Further, the program of claim 12 is characterized in that, in the program of claim 10 or 11, the spatial feature amount is a fractal dimension calculated based on the face image of the subject.

In addition, the method for determining a stress coping method according to claim 13 is characterized by comprising: a biometric information acquisition step of acquiring biometric information of the subject in a non-contact state; and a determination step based on the biometric information and a predetermined response pattern , determine the stress coping mode of the examinee, and the response mode is determined by the hemodynamic parameters.

The stress coping mode determination method acquires the biological information of the subject in a non-contact state, and determines the stress coping mode of the subject based on the response mode determined by the biological information and the hemodynamic parameters.

In addition, the learning device of claim 14 is characterized by comprising: a learning data storage unit that stores a plurality of learning face images labeled corresponding to the response patterns determined by the hemodynamic parameters; and a feature value extraction unit that uses learning completed The model extracts the spatial feature quantity of the face image of the subject from the face image for learning; the feature quantity learning part is based on the extraction result obtained by the feature quantity extraction part and the face for the study as its extraction object. The relationship between the labels of the image annotations, and the network parameters of the learned model are changed so that the extraction accuracy of the spatial feature obtained by the feature extraction unit becomes high.

The learning device stores a plurality of facial images for learning that are labeled corresponding to the response patterns determined by the hemodynamic parameters, uses the learned model to extract the spatial feature quantity of the facial image of the subject from the facial images for learning, and based on the Regarding the relationship between the extraction result and the label attached to the learning facial image that is the extraction target, the network parameters of the learned model are changed so that the extraction accuracy of the spatial feature amount of the facial image of the subject is improved.

Further, in the learning apparatus of claim 15, in the learning apparatus of claim 14, the spatial feature amount is a fractal dimension calculated based on a face image of the subject.

In addition, the learning method of the scheme 16 is characterized by comprising: a data storage step for learning, storing a plurality of face images for learning with labels corresponding to the response patterns determined by the hemodynamic parameters; and a feature extraction step, using the learning completed The model extracts the spatial feature quantity of the face image of the subject from the face image for learning; the feature quantity learning step is based on the extraction result obtained by the feature quantity extraction part and the face for the study as its extraction object. The relationship between the labels of the image annotations, and the network parameters of the learned model are changed so that the extraction accuracy of the spatial feature obtained by the feature extraction unit becomes high.

The learning method stores a plurality of face images for learning with labels corresponding to the response patterns determined by the hemodynamic parameters, and uses the learned model to extract the spatial feature quantity of the face image of the subject from these face images for learning, based on The relationship between the extraction result and the label attached to the learning facial image, which is the extraction target, changes the network parameters of the learned model so as to improve the extraction accuracy of the spatial feature of the subject's facial image.

Further, the learning method of claim 17 is characterized in that, in the learning method of claim 16, the spatial feature amount is a fractal dimension calculated based on a face image of the subject.

In addition, the program of claim 18 is for causing the computer to function as a means for learning the spatial feature amount of the face image, and it is characterized by having a data storage step for learning that stores a label corresponding to the response pattern determined by the hemodynamic parameter. A plurality of learning facial images; Feature extraction step, using the learning completed model to extract the spatial feature of the subject's facial image from the learning facial image; Feature learning step, based on the feature extraction step by the The relationship between the obtained extraction result and the label labeled with the learned facial image as its extraction object, and the network parameters of the learned model are changed so that the extraction of the spatial feature obtained by the feature extraction step is performed. Accuracy becomes higher.

The program is installed and executed in one or a plurality of computers working in cooperation with each other, so that the system constituted by the one or more computers functions as a learning device that stores responses corresponding to patterns determined by hemodynamic parameters A plurality of labeled facial images for learning are used to extract the spatial feature quantity of the facial image of the subject from these facial images for learning by using the learned model, based on the extraction result and the labeling of the facial image for learning that is the extraction object. In relation to the labels, the network parameters of the learned model are changed so that the extraction accuracy of the spatial feature data of the face image of the subject is improved.

Furthermore, the program of claim 19 is characterized in that, in the program of claim 18, the spatial feature amount is a fractal dimension calculated based on the face image of the subject.

In addition, the learned model of claim 20 is characterized in that a plurality of face images for learning labeled corresponding to the response patterns determined by the hemodynamic parameters are used for training data, and the spatial features of the face images of the subject are analyzed by The learned model is generated by performing machine learning on the quantity.

The learned model takes the subject's face image as an input, and outputs the spatial feature amount of the subject's face image.

Further, the learned model of claim 21 is characterized in that, in the program of claim 18, the spatial feature amount is a fractal dimension calculated based on a face image of the subject.

Invention effect

According to the stress response mode determination system of the first solution, the biological information of the subject can be acquired in a non-contact state, and the subject can be determined in a non-contact state based on the biological information and the response mode determined by the hemodynamic parameters. Therefore, the type of stress felt by the subject can be grasped without imposing restrictions on the subject's actions.

Usually, the term "stress" is widely used, and in psychology, "stress" refers to "the general term for non-specific responses to external stimuli (stressors) acting on an organism" ( Psychologist Dr. Han Selye).

It is known that there are various stressors based on the survival of human beings. In recent years, there have been many stressors in the social environment in particular, and they have various effects on living organisms. In some cases, stressors can also be the cause of disease.

It is known, however, that stress is not all bad for the human being, and that even under presumed stressful situations, organisms activate when trying to cope with stress to beneficial effects. From such a point of view, Dr. Sells clearly expressed the idea that stress can be good stress (enstress) or bad stress (distress) depending on the difference in the recipient's biological condition or the degree of stress.

Therefore, from such a psychological point of view, when considering stress, it is not possible to grasp all stress as bad, and it is necessary to classify and consider the types of stress in living organisms as described above. In addition, from such a viewpoint, it is possible to actively consider how to manage stress in various modern social activities from the perspective of sociality and productivity, such as whether to improve production efficiency and work efficiency.

However, as described above, techniques and patented inventions for grasping the stress state of the subject are known, and the conventional methods can only roughly grasp the stress felt by the subject, and cannot perform a method based on the subject's feeling. A qualitative analysis of the type of stress received.

According to the present invention, by classifying and grasping the type of stress according to the subject, the relationship between the stressor and the stress response of the subject can be grasped in more detail. As a result, it is possible to more accurately analyze and study the relationship between people, such as the social environment, which is a stressor, and it can be applied to various social environmental fields to solve problems, and can be applied to various industrial fields. Improve production efficiency and promote production activities.

According to the stress coping mode determination system of claim 2, it is possible to determine in detail and based on the biological information of the subject and the response pattern determined by any of the mean blood pressure, heart rate, cardiac output, stroke volume, and total peripheral vascular resistance. Accurately determine how the subject copes with stress.

According to the stress response mode determination system of claim 3, the face image of the subject can be acquired in a non-contact state, and the continuous sphygmomanometer can be The subject's response to stress can be quickly determined without restraint and without placing a physical burden on the subject.

According to the pressure coping mode determination system of scheme 4, the facial thermal image or facial visual image of the subject can be acquired in a non-contact state, and based on the facial thermal image or facial visual image and the response mode determined by the hemodynamic parameters , according to psychophysiology to determine the subject's stress coping style.

According to the stress coping mode determination system of claim 5, it is possible to easily and accurately determine the stress coping mode of the subject based on the response pattern determined by the state of the specific part of the subject's face and the hemodynamic parameters.

According to the stress coping mode determination system of claim 6, the biological information of the subject can be acquired in a non-contact state, and it is determined based on the biological information whether the stress coping mode of the subject is "active coping" or "passive coping" " and "no response" which response mode.

Therefore, since the stress response methods can be classified into the above-mentioned three response modes of "active response", "passive response", and "no response", analysis based on the response mode can be performed, and the analysis results can be applied to various production fields. Personnel management business, quality management business, etc., contribute to the quality improvement of various businesses.

According to the stress coping method determination system of the solution 7, it is possible to store the spatial feature quantity corresponding to "active coping", the spatial characteristic quantity corresponding to "passive coping", and the spatial characteristic quantity corresponding to "no coping", based on the subject's Based on the biological information and each spatial feature amount, it is determined whether the stress coping mode of the subject shows which response mode of "active coping", "passive coping", and "no treatment".

According to the stress response method determination system of claim 8, the network parameters of the learned model can be changed so that the extraction accuracy of the spatial feature data of the face image of the subject can be improved.

According to the stress coping mode determination system of claim 9, since the spatial feature quantity can be quantified with higher accuracy using fractal dimension, the stress coping mode of the subject can be determined with high accuracy in a non-contact state.

According to the program of claim 10, it is possible to use one computer or a plurality of computers working in cooperation with each other to realize a system based on the spatial feature quantity corresponding to "active coping", the spatial characteristic quantity corresponding to "passive coping", and the corresponding "no coping" and the subject's facial image, it is determined which response mode of "active coping", "passive coping" and "no coping" is shown in the stress coping mode of the subject.

According to the program of claim 11, it is possible to use one computer or a plurality of computers working in cooperation with each other to realize the following system: store a plurality of face images for learning labeled corresponding to "active coping", "passive coping" and "no coping", respectively, and use The learned model extracts the spatial feature of the face image for learning, and based on the relationship between the extraction result and the label labeled with the face image for learning that is the object of its extraction, the network parameters of the learned model are changed to make the extraction accuracy of the spatial feature variable. high, and the extracted spatial feature quantities are stored.

According to the program of claim 12, since the spatial feature quantity can be quantified with higher accuracy by using fractal dimension, it is possible to realize the stress response that can determine the subject's stress response with high accuracy in a non-contact state using one computer or a plurality of computers working in cooperation with each other. way system.

According to the method for determining the stress response mode of claim 13, the biological information of the subject can be acquired in a non-contact state, and the subject can be determined in a non-contact state based on the biological information and the response mode determined by the hemodynamic parameters Therefore, the type of stress felt by the subject can be grasped without imposing restrictions on the subject's actions.

According to the learning device of claim 14, the network parameters of the learned model can be changed so that the extraction accuracy of the spatial feature data of the face image of the subject can be improved.

According to the learning device of claim 15, since the spatial feature amount can be quantified with higher accuracy using fractal dimension, the network parameters of the learned model can be changed so that the extraction accuracy of the spatial feature amount of the face image of the subject becomes higher. higher.

According to the learning method of claim 16, the network parameters of the learned model can be changed so that the extraction accuracy of the spatial feature data of the face image of the subject can be improved.

According to the learning method of claim 17, since the spatial feature amount can be quantified with higher accuracy by using fractal dimension, the network parameters of the learned model can be changed so that the extraction accuracy of the spatial feature amount of the face image of the subject can be improved. higher.

According to the program of claim 18, by installing and executing the program on one or a plurality of computers working in cooperation with each other, it is possible to change the network parameters of the learned model so as to improve the extraction accuracy of the spatial feature amount of the face image of the subject. learning device.

According to the program of claim 19, since the spatial feature quantity can be quantified with higher accuracy by using fractal dimension, the network parameters of the learned model can be changed by installing and executing the program on one or a plurality of computers working in cooperation with each other. In order to improve the extraction accuracy of the spatial feature amount of the face image of the subject.

According to the learned model of claim 20, by inputting the face image of the subject to the learned model, the spatial feature amount of the face image of the subject can be extracted.

According to the learned model of claim 21, since the spatial feature amount can be quantified with higher accuracy using fractal dimension, the subject can be extracted with high accuracy by inputting the face image of the subject into the learned model. The spatial feature quantity of the facial image.

Description of drawings

FIG. 1 is a block diagram of an embodiment of a stress response mode determination system according to the present invention.

FIG. 2(A) is an explanatory diagram conceptually illustrating a group of face images for learning to which a label of “active coping” is attached. (B) is an explanatory diagram conceptually illustrating a group of face images for learning to which the label "passive coping" has been attached. (C) is an explanatory diagram conceptually illustrating a group of face images for learning to which a label of "no response" is attached.

FIG. 3 is a flowchart showing the processing content of the determination device constituting the stress response mode determination system of FIG. 1 .

FIG. 4 is a flowchart showing the processing content of the learning device constituting the stress coping mode determination system of FIG. 1 .

FIG. 5 is a table showing the hemodynamic pattern response of the conventional study of Experimental Example 1. FIG.

FIG. 6 is a conceptual diagram showing a measurement system.

FIG. 7 is a conceptual diagram illustrating the problem of mirror drawing.

FIG. 8 is a graph showing time-series changes in MBP (Mean Blood Pressure).

FIG. 9 is a schematic diagram showing the structure of a CNN (Convolutional Neural Network).

FIG. 10 is a table showing the filter size, step size, and number of filters of the convolutional layer.

FIG. 11 is a table showing the filter size and step size of the pooling layer.

FIG. 12 is a diagram showing the characteristics of the face in each case of active coping, passive coping, and no coping of each subject, and showing the thermal image and the feature map of the face in comparison.

FIG. 13 is a conceptual diagram showing the experimental protocol of Experimental Example 2. FIG.

FIG. 14 is a graph showing the relationship between the degree of preference and the degree of concentration.

FIG. 15 is a diagram showing an arrangement state of electroencephalogram electrodes on a subject and a measurement system.

FIG. 16 is a table showing changes in the multi-faceted emotion scale for "positive" content.

FIG. 17 is a table showing changes in the multi-faceted emotion scale for "negative" content.

FIG. 18 is a table showing changes in the multi-faceted emotion scale for "horror (focus)" content.

FIG. 19 is a table showing changes in the multi-faceted emotion scale for "horror (non-focused)" content.

FIG. 20 is a table showing changes in subjective psychological indicators for "positive" content.

FIG. 21 is a graph showing changes in subjective psychological indicators for "negative" content.

FIG. 22 is a table showing changes in subjective psychological indicators for the content of "horror (focus)".

FIG. 23 is a graph showing the variation of the subjective psychological index with respect to the content of "horror (unfocused)".

FIG. 24 is a table showing time-series changes in physiological indicators for viewing of “positive” and “negative” content.

FIG. 25 is a table showing the evaluation of the physiological index of each content.

FIG. 26 is a graph showing time-series changes in physiological indicators for viewing of “horror (focus)” and “horror (non-focus)” content.

FIG. 27 is a graph showing the relationship between preference for TV content and viewing method.

FIG. 28 is a conceptual diagram showing the experimental protocol of Experimental Example 3. FIG.

FIG. 29 is a conceptual diagram illustrating the experimental protocol.

FIG. 30 is a table showing the structure of a neural network for estimating "excitement-sedation", "stress coping style", and "favorite".

FIG. 31 is a diagram illustrating a method of extracting feature vectors.

FIG. 32 is a table showing evaluations of subjective psychological indicators for each content.

FIG. 33 is a table showing the evaluation of the physiological index of each content.

FIG. 34 is a table showing the positive discrimination rate of preference for TV content and viewing method.

FIG. 35 is a graph showing the positive discrimination rate of preference for TV content and viewing method.

FIG. 36 is a graph showing the temporal changes of the actual measured value and the estimated value of the “excited-sedated” state.

Fig. 37 is a graph showing the estimation error of the "excited-sedated" state.

FIG. 38 is a graph showing the relationship between the actual measured value and the estimated value of the “excited-sedated” state.

FIG. 39 is a flowchart illustrating a method of obtaining a fractal dimension as a spatial feature quantity.

FIG. 40 is an explanatory diagram showing an example of the clustering process in FIG. 39 .

FIG. 41 is an explanatory diagram showing an example of the image extraction process and the edge extraction process in FIG. 39 .

FIG. 42 is an explanatory diagram showing an example of the fractal analysis process in FIG. 39 .

Detailed ways

Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[constitute]

The stress coping method determination system 100 according to the embodiment shown in FIG. 1 includes a biological information acquisition device (biological information acquisition unit) 110 , a determination device (determination unit) 120 , and a learning device (machine learning unit) 130 .

The biological information acquisition device 110 is a device for acquiring biological information of the subject P in a non-contact state.

The most suitable example of the biological information can be the face image IF. In the following description, the case where the face image IF is used as the biological information will be described.

The face image IF can be a thermal image of the face or a visible image of the face. In the case where the face image IF is a face thermal image, an infrared thermal imager is used as the biological information acquisition device 110 . When the face image IF is a visible face image, a so-called camera, which is a visible image capturing device, is used as the biological information acquisition device 110 .

As described above, the "face visible image" is a color image obtained by photographing the face of a subject with a generally widely used camera, that is, a device having an optical system for imaging and photographing an image. Moreover, the "facial thermal image" is an image which analyzes the infrared rays radiated from the face of a subject, and shows a heat distribution as a graph, and is an image image|photographed by an infrared thermal imager.

In this case, in the case of the image captured by the camera, the image is formed by visible light (wavelengths of 380 nm to 800 nm), while the thermal distribution image obtained by the infrared thermal imager is formed by infrared rays (800 nm or more). Therefore, in any case, there is only a difference in wavelength, so either an infrared thermal imager or a general camera can be used as the biological information acquisition device 110 .

The determination means 120 can be realized by installing and executing the program of the present invention in a general-purpose computer.

The determination device 120 is a device having a function of determining the stress coping style of the subject based on the face image IF acquired by the biological information acquisition device 110 and a predetermined response pattern. The response mode includes three modes consisting of "active response" (mode I), "passive response" (mode II), and "no response" (mode III). The response mode is determined by hemodynamic parameters. Hemodynamic parameters include multiple parameters in mean blood pressure (MBP), heart rate (HR), cardiac output (CO), stroke volume (SV) and total peripheral vascular resistance (TPR).

The determination device 120 includes a determination feature amount storage unit 121 , a specific site reaction detection unit 122 , and a response pattern determination unit 123 .

The feature value storage unit 121 for determination is a functional block that stores a spatial feature value corresponding to "active response", a spatial feature value corresponding to "passive response", and a spatial feature value corresponding to "no response". The spatial feature amount stored in the feature amount storage unit 121 for determination is the spatial feature amount extracted by the learning device 130 . When the face image IF is a face thermal image, as an example of the spatial feature quantity, the temperature distribution of the face skin can be exemplified.

The specific part reaction detection unit 122 is a functional block that detects the pressure response of the anatomical specific part of the face of the subject P included in the face image IF. An anatomically specific site is one or more sites determined from an anatomical viewpoint as sites with little individual differences. As an example of an anatomically specific part, the roof of the nose can be cited.

The response mode determination unit 123 determines, based on the stress response detected by the specific site reaction detection unit 122 and the respective spatial feature values stored in the determination feature value storage unit 121, that the stress response mode of the subject P is to indicate "active". A functional module that responds to which of the "response" (mode I), "passive response" (mode II) and "no response" (mode III) mode.

The learning device 130 includes a learning data storage unit 131 , a feature value extraction unit 132 , and a feature value learning unit 133 . The learning device 130 is realized by installing the program of the present invention in a general-purpose computer and executing it.

The learning data storage unit 131 is a functional module that stores a plurality of learning face images LG labeled with corresponding labels corresponding to “active response”, “passive response”, and “no response”, respectively. The face image LG for learning is conceptually illustrated in FIG. 2 . LGA1 , LGA2 , LGA3 , . . . shown in FIG. 2(A) are face image groups for learning to which the “active response” label is attached. LGB1, LGB2, LGB3, . . . shown in FIG. 2(B) are face image groups for learning to which a “passive coping” label is attached. In addition, LGC1, LGC2, LGC3, . . . shown in FIG. 2(C) are face image groups for learning to which a label of “no response” is attached.

The feature extraction unit 132 is a functional block for extracting the spatial feature of the face image from the learning face image LG using the learned model 134 . Using a plurality of learning face images LG labeled corresponding to the response patterns determined by the hemodynamic parameters as training data, the processing is performed by analyzing the spatial feature amount of the face image of the subject P included in the learning face image LG. The learned model 134 is generated by machine learning.

The feature value learning unit 133 changes the network parameters of the learned model 134 based on the relationship between the extraction result of the feature value extracting unit 132 and the label attached to the learning face image LG as the extraction target so that the space of the feature value extracting unit 132 A functional module that improves the extraction accuracy of feature quantities.

[action]

Next, the flow of processing in the determination device 120 and the learning device 130 of the stress coping mode determination system 100 configured as described above will be described in accordance with the flowcharts of FIGS. 3 and 4 .

As shown in FIG. 2 , the determination device 120 executes determination feature storage processing S1 , specific site reaction detection processing S2 , and response pattern determination processing S3 .

The feature value storage processing S1 for determination is to store the spatial feature values extracted by the learning device 130 , namely, the spatial feature value corresponding to the “active response”, the spatial feature value corresponding to the “passive processing”, and the spatial feature value corresponding to the “no response”. The processing stored in the feature value storage unit 121 for determination.

The specific part reaction detection process S2 is a process of detecting the pressure response of the anatomical specific part of the face of the subject P included in the face image IF captured by the biological information acquisition device 110 .

The response mode determination process S3 is based on the stress response detected in the specific site reaction detection process S2 and the respective spatial feature amounts stored in the feature amount storage unit 121 for determination, to determine that the stress response mode of the subject P is to indicate "active". What kind of response mode among "response", "passive response" and "no response" is handled.

As shown in FIG. 4 , the learning device 130 executes the learning data storage process S11 , the feature amount extraction process S12 , and the feature amount learning process S13 .

The learning data storage process S11 is a process of storing, in the learning data storage unit 131 , a plurality of learning face images LG labeled with corresponding labels corresponding to “active response”, “passive response”, and “no response”, respectively.

The feature extraction process S12 is a process of extracting the spatial feature of the subject's face image included in the learning face image LG from the learning face image LG using the learned model 134 .

The feature amount learning process S13 is to change the network parameters of the learned model 134 based on the relationship between the extraction result of the feature amount extraction process S12 and the label attached to the learning face image LG as the extraction target so that the feature amount extraction process S12 is performed. Processing to improve the extraction accuracy of the amount.

[spatial feature amount]

In this embodiment, the fractal dimension calculated based on the face image IF can be used as the spatial feature amount. By using the fractal dimension, the spatial feature quantity can be easily and accurately quantified.

The method of obtaining the fractal dimension as the spatial feature quantity is arbitrary. In the processing flow illustrated in FIG. 39 , the fractal dimension is obtained by executing a series of processing including clustering processing S21 , image extraction processing S22 , edge extraction processing S23 , and fractal analysis processing S24 .

The clustering process S21 is a process of clustering the temperature distribution of the face image IF. The method of clustering is arbitrary, and as a method suitable for the present embodiment, a fuzzy C-means (Fuzzy-c-means) method can be exemplified. The fuzzy C-means method is not a method that considers whether a data group belongs to a cluster (cluster) or not. , a method in which data is assumed to belong to a plurality of clusters to a certain degree, and the degree of belonging (attribution) of the data to a cluster is shown vaguely. FIG. 40 shows an example of clustering the input image (face image IF) by setting the number of clusters to 12. In the example of FIG. 40 , cluster 1 belongs to the lowest temperature distribution, and cluster 12 belongs to the highest temperature distribution.

The image extraction process S22 is a process of extracting a clustered image of the temperature distribution of a temperature region having a predetermined temperature or higher from the plurality of clustered images obtained by the clustering process S21 . In the example of FIG. 41 , among the images of clusters 1 to 12 illustrated in FIG. 40 , images of clusters 8 to 12 , which are images of attribution degrees including the temperature of the face region, are extracted.

The edge extraction process S23 is a process of detecting an edge portion of the image extracted by the image extraction process S22, and generating an edge figure composed of line segments represented by the edge portion. The method of edge detection is arbitrary, and as a method suitable for this embodiment, the Canny method can be exemplified. The canny method is a noise removal process performed by a Gaussian filter, a luminance gradient (edge) extraction process performed by a Sobel filter, a non-maximum value suppression process that removes portions other than the portion where the intensity of the edge reaches a maximum, and A method of detecting an edge by hysteresis threshold processing using a threshold value of hysteresis (Hysteresis) to determine whether or not the edge is correct. In the example of FIG. 41 , the edge portions of the images of clusters 8 to 12 are detected by the canny method, and an edge pattern composed of line segments represented by the edge portions is generated.

The fractal analysis process S24 is a process of obtaining the fractal dimension of the edge figure generated by the edge extraction process S23. Fractal dimension is an index that quantitatively expresses the self-similarity and complexity of graphics and phenomena, and is generally a non-integer value. The method of fractal analysis is arbitrary, and as a method suitable for this embodiment, a box counting method can be exemplified. The box-counting method is a method in which, when a figure to be analyzed is divided into square boxes (lattice-like), a straight line is approximated on a logarithmic hyperbola from the relationship between the size of the box and the total number of boxes containing the figure. The absolute value of the slope to find the fractal dimension.

The calculation formula of the fractal dimension (D) is represented by the following formula. r is the size of the box and N(r) is the number of boxes.

[Number 1]

FIG. 42 shows an example of calculation of the fractal dimension of the edge pattern of the cluster 12 in FIG. 41 . In this example, with respect to the edge pattern of cluster 12, r was varied in the range of 2 to 128, and r and N(r) were plotted on a logarithmic hyperbola to obtain a value of 1.349.

[Function and effect]

In the stress coping mode determination system 100 configured as described above, the face image IF of the subject P is captured by the biological information acquisition device 110 . The captured face image IF is input to the determination device 120 . The determination means 120 is based on the input facial image IF and the response pattern (patterns I, II) determined by the hemodynamic parameters, that is, any of the mean blood pressure, heart rate, cardiac output, stroke volume, and total peripheral vascular resistance. and III), it is determined which of the modes I, II, and III is shown as the stress coping mode of the subject P.

Therefore, according to the stress coping mode determination system 100 , the stress coping mode of the subject P can be determined in a non-contact state based on the facial image IF of the subject P, so that the subject P can be prevented from restricting his actions. The type of stress felt by the subject P is grasped under the circumstances.

In addition, the stress coping mode determination system 100 determines the stress coping of the subject P based on the reactions and response patterns (patterns I, II and III) of the anatomical specific parts of the face included in the face image IF of the subject P Way. An anatomically specific site is one or more sites determined from an anatomical viewpoint as sites with little individual differences.

Therefore, according to the stress coping mode determination system 100, a general system for determining the stress coping mode can be constructed.

In addition, in the stress coping mode determination system 100, the learning device 130 extracts a feature amount corresponding to "active coping", a feature amount corresponding to "passive coping", and a feature amount corresponding to "no coping". The learning device 130 stores a plurality of face images LG for learning that are labeled corresponding to “active coping”, “passive coping”, and “no coping”, respectively, and extracts the face image space from these learning face images LG using the learned model 134 The feature quantity, based on the relationship between the extraction result and the label attached to the learning facial image LG as the extraction target, changes the network parameters of the learned model LG so as to improve the extraction accuracy of the spatial feature quantity of the facial image . As the learning of the learned model LG progresses, the extraction accuracy of the spatial feature data of the face image is improved, and the accuracy of the spatial feature data stored in the determination feature data storage unit 121 is also improved.

Therefore, according to the stress coping mode determination system 100 , the determination accuracy of the stress coping mode can be improved as the learning of the learned model LG in the learning device 130 progresses.

In addition, according to this stress coping mode determination system, the determination accuracy of the stress coping mode can be further improved by quantifying the spatial feature quantity using the fractal dimension.

In addition, this invention is not limited by the said embodiment. For example, in the above-described embodiment, the stress coping mode determination system 100 includes the learning device 130, but the learning device 130 may be omitted. When the learning device 130 is omitted, the spatial feature amount extracted or generated by means other than the learning device 130 is stored in the determination feature amount storage unit 121 of the determination device 120 .

<Experimental example 1>

Hereinafter, an experimental example for collecting data of feature amounts stored in the feature amount storage unit 121 for determination of the present embodiment will be described.

1. Experimental method

The experiment was performed in a shielded room at 25.0±1.5°C, and the subjects were 8 healthy adult males aged 18 to 21. The measurement system is shown in FIG. 6 . The subject was seated, and a measurement cuff of a continuous blood pressure and hemodynamic dynamic measuring device (Finometer model 2, manufactured by Finapres Medical Systems B.V.) was worn on the second joint of the left middle finger.

As hemodynamic parameters, mean blood pressure (MBP), heart rate (HR), cardiac output (CO), total peripheral vascular resistance (TPR) were measured. In order to measure the thermal image of the face, an infrared thermal imager (TVS-200EX, AVIONICS, Inc.) was set at a position of 1.2 m in front with a viewing angle capable of measuring the entire face, and the shooting interval was set to 1 fps. When measuring the facial skin temperature, sit on a chair with a backrest, and use a projector to project the computer screen at a distance of 1.55m to the wall.

The experiment consisted of the time division of quiet eyes closed for 60s before the task (Rest1), active task for 60s (Task1), no response task for 60s (Task2), passive task for 60s (Task3), and quiet eyes closed for 60s after the task (Rest2). Mental arithmetic was conducted in the active task, mirror writing task was conducted in the passive task, and eyes closed quietly in the stress-free coping task. The mental arithmetic task is a 2-digit addition every 4 seconds, performed on a computer screen. Every time the mental arithmetic is not told the right or wrong. In addition, in the mirror writing task, the subject is instructed to use the mouse with the right hand and reproduced on the computer by passing between the lines of the star pattern (see FIG. 7 ) displayed on the screen. The movement of the mouse and the movement of the cursor on the screen are reversed up and down, left and right.

Displays the cursor and the track of the cursor on the star graph, and returns the cursor to the original position when out of bounds and erases all tracks. The wrapping start position is the uppermost part of the star. Quiet and closed eyes are used as a problem that cannot be dealt with. Normalization was performed by subtracting the mean value of Rest1 in the hemodynamic parameters as the baseline.

2. Analysis method

2-1 Discrimination of hemodynamic parameters

The time-series changes in the normalized MBP of the subject A are shown in FIG. 8 . Regarding each hemodynamic parameter, the range of the value greater than or equal to the baseline +2σ (σ: standard deviation of each hemodynamic parameter in Rest1) is defined as “+”, and the range of the value of the baseline −2σ or less is defined as “+”. defined as"-". Based on the pattern responses in Table 1, each hemodynamic parameter was judged as "pattern I" (active response) and "pattern II" (passive response), and a case that did not meet either of these parameters was judged as "no response". In addition, based on the discrimination of hemodynamic parameters, the thermal images captured during Tasks 1 to 3 were labeled.

2-2 Creation of input data

In order to use the labeled thermal images in machine learning, parts of the face were cropped at 151×171 pixels and grayscaled to create a thermal image of the face. In addition, in order to make the number of face thermal images (input data) after labeling consistent with each stress response, random cropping, white Gaussian noise addition, and contrast adjustment are performed to expand the data.

2-3 Machine Learning Using CNN

In this experiment, a convolutional neural network (CNN) was used to construct an individual discriminant model of stress coping styles based on facial skin temperature distribution, and feature quantities related to stress coping styles were extracted.

The configuration of the CNN consists of three convolutional layers for extracting feature quantities, three pooling layers, and one fully connected layer for discrimination. The structure of the CNN is shown in FIG. 9 , and the parameters of the filters of the convolution layer and the pooling layer are shown in FIGS. 10 and 11 . Based on the feature map that is the weight of the convolutional layer of the CNN, the feature analysis related to the stress response method is performed.

3. Results and Research

Fig. 12 shows the feature map of the second convolutional layer when the facial thermal image of each stress response method is input to the CNN for each subject.

The results of observing the feature map of each coping style in the subjects showed that, especially in subject B, the characteristic parts displayed between the stress coping styles were different, such as active coping and stress-free coping. Features were identified on the left cheek, and features were identified on the right cheek when responding passively. As a result of observing the feature map among the subjects, most of the subjects showed features in the nose, but it was confirmed that there were individual differences in the parts where the features were expressed in each subject.

The main reason for the identification of individual differences in the characteristic expression sites of each subject is considered to be differences in the structures of blood vessels or fat. However, it is considered that by identifying the anatomically meaningful feature parts, it is possible to build a general model for stress response discrimination.

4. Summary

In this experiment, by using CNN, we tried to discriminate stress coping styles based on facial skin temperature distribution and extract features related to stress coping styles. As a result, the characteristic distribution of the stress coping style in the facial skin temperature distribution changed, and it was confirmed that there were individual differences in the characteristic distribution related to the stress coping style among individuals.

references

[1] Yukio Sawada: New Physiological Psychology (edited by Hiro Miyata), Chapter 10, Kitaoji Study, pp.172-195, (1998)

[2] Yuichiro Nagano: "Cardiovascular Responses in Competitive Mirror Writing Projects", Physiological Psychology and Psychophysiology, Vol.22, No3, pp.237-246, (2004)

[3] Shizuo Bando, Yutoshi Asano, Akio Nozawa: "Analysis of Physiological Influences Related to Display Media of Reading", Journal of Electrical Engineering Society C (Electronics, Information, and Systems Department), Vol.135, No.5 , pp.520-525, (2015)

[4] Takuya Watanuki, Akio Nozawa: "Analysis of TV viewing methods based on stress coping methods", Journal of Electrical Engineering Society C (Electronics, Information, and Systems Department), Vol.134, No.2, pp.205 -211, (2014)

[5] Nikki-kyu, Nozawa Akio, Mizuno Tota, Iide Hideto: "Physiological and Psychological Evaluation of Mental Workload in Time-critical Situations", Papers of the Electrical Society C (Electronics, Information, and Systems Departments), Vol.127, No.7, pp.1000-1006, (2007)

[6] Hiroki Ito, Shizuka Bando, Kosuke Oiwa, Akio Nozawa: "Evaluation of Variations in Autonomic Nervous System's Activity During the Day Based on Facial Thermal Images Using Independent Component Analysis ", IEEJ Transactionson Electronics, Information and Systems, Vol.138, No.7, pp.1-10, (2018)

[7] Kenta Matsumura, Yuki Sawada: "Cardiovascular Responses When Two Mental Arithmetic Projects Are Completed", Psychological Research, Vo79, No.6, pp.473-480, (2008)

In addition, two examples of experimental studies conducted by the inventors of the present invention, which are basic studies of the stress response mode determination system of the present invention, are shown below as an application example (example) of the present invention.

These experimental examples are not experimental examples of "acquiring the biological information of the subject in a non-contact state" as in the present invention, but were conducted by using a continuous sphygmomanometer to make the subject wear a finger cuff, but the same as the present invention Based on the hemodynamic parameters, it analyzes, classifies and evaluates the physiological and psychological states of the viewers while watching TV video content. Therefore, the present invention can be applied to, for example, determination of the stress pattern of the subject in such a case.

<Experimental example 2>

the subject of the experiment

Nowadays, people are surrounded by a large number of information devices. Among them, the television has been recognized as a core existence of news media or entertainment media installed in the living room of a family or the like since its invention. However, with the popularization of miniaturized and high-performance information communication equipment and the expansion of the information network environment brought about by the development of IT technology in recent years, the existence of information media has undergone a major transformation, and the way of viewing television has also undergone a major transformation.

In the article, Nishigaki pointed out that always turning on the TV while doing chores or operating a mobile phone, etc., is obviously far from a "concentrated viewer" ⁽¹⁾ . However, if you can grasp the way of watching this transformation, you can make TV with new added value, such as adjusting emotions, moods, and actions in the desired direction through TV content design.

Fujiwara and Saito conducted a public opinion survey, and Ohno conducted a questionnaire survey. Examples of the reasons for watching TV include "passing the time", "to enjoy interesting programs", "to improve cultural accomplishment and knowledge", "out of habit"","To escape the annoying reality", "To change the mood", "To be the background music", "To reunite the family" and other reasons ⁽²⁾⁽³⁾ .

In addition, Takahashi et al. classified the time of watching TV into "breakfast", "commute", "doing housework", "relaxing at home on weekdays", etc. ⁽⁴⁾ . In addition, Tomozune et al. classified the way of watching TV into "watching exclusively", which is a state where "both mind and body are focused on TV", and "viewing exclusively", which is a state where "I want to watch this program, but I will watch it and do other things at the same time". Simultaneous & dedicated switching viewing", "There is no particular content to watch, but the body is focused on the TV" such as "relaxation time viewing" and so on.

As such, it is clear that there are many ways to watch, depending on factors such as time, place, mood, etc. The preference for TV images is also a reason for the transformation of viewing methods. However, there are no studies that categorize viewing patterns by preference. In addition, almost all of these studies that discriminate viewing patterns have used psychological responses such as public opinion surveys or post-mortem introspective evaluations, and there have been no studies that have used physiological responses to classify viewing patterns.

Since the images and sounds of TV are visual and auditory stimuli to the living body, they can be regarded as stressors themselves. Stressors prompt the organism to respond to stress, which brings about physiological and psychological changes in response to stress. If various contents of television are regarded as stressors, it can be expected that the response to the contents, that is, the way of viewing, varies in response to physiological stress. Nomura et al. will classify the corresponding states of e-learning content by cardiovascular system indicators ⁽⁶⁾ .

Therefore, in this study, the cardiovascular system measurement using a continuous sphygmomanometer, non-contact facial skin temperature measurement using an infrared thermal imaging device, and digital biometric The electrocardiogram and brain wave measurements of the body amplifier were used to evaluate the activity of the central nervous system and the activity of the autonomic nervous system. In particular, an analysis of the viewing style when watching TV images was performed by using the stress coping style and preference for TV images using hemodynamic parameters.

2. Factor extraction experiment

Considering that the viewing mode is affected by the video content and personal preference, the classification of the factors is carried out through preliminary experiments.

<2.1> Experimental conditions

The subjects were 20 healthy students (age: 19-22, average age: 21.1, 11 males, 9 females) who were studying in the university. The subjects were fully explained to the subjects orally and in writing in advance about the content of the experiment, the purpose, and the subjects of the investigation, and their consent to the experimental assistance was confirmed by signature. Measurements were performed in a shielded room with no convection at room temperature of 26.0±1.6°C, with no temperature input from outside.

The subject watched 5 minutes of 10 video contents (news, sports, documentaries, music, horror, comedy, dramas, cartoons, cooking, shopping). The video content is played by a DVD player (SCPH-3900, SONY). The video content viewed will not be released until the subject has viewed it. In addition, in order to eliminate the order effect, the order of the video content to be viewed is set to a random order.

<2.2> Experimental procedure

As shown in FIG. 13 , this experiment consisted of 5 minutes of video viewing and 1 minute of quiet eye-closed states R1 and R2 before and after it. In addition, preference and immersion evaluation of various video contents were performed by Visual Analogue Scale (hereinafter abbreviated as VAS) and immersion survey.

<2.3> Evaluation method

At the end of the experiment, the VAS was used to measure the subjective perceived quantity of 'liking'. In VAS, a pair of adjectives are arranged at both ends of a line segment, and the examinee selects any position on the line segment, whereby the psychophysical quantity of the examinee can be measured. Set the sentences configured at both ends of the ruler to "very annoying" - "very fond". In addition, the immersion in the video was measured by a 5-point scale (1: not immersed at all - 5: quite immersive).

<2.4> Result and research of factor extraction experiment

The left side of FIG. 14 shows the average of all subjects in the preferences and immersion of 10 kinds of content. The error bars in the figure represent the standard errors of liking and immersion. On the right side of FIG. 14 , among the 10 kinds of contents, especially the news and horror with large individual differences are shown as the preferences of the 20 subjects to give a sense of immersion. In FIG. 14 , if the preference and immersion of each content are viewed from the left, the average preference and immersion are in a proportional relationship. In FIG. 14, N=20.

Regarding horror, there was a peculiar result of low preference but high immersion. It can be considered that this is due to the so-called "the more I am afraid, the more I want to watch" because I hate it but I want to watch it out of curiosity. As can be seen from the right side of FIG. 14 , since each subject has different preferences and immersion, there are large individual differences. Therefore, in the physiological measurement experiment, based on the above-mentioned factor classification experiment, each subject is divided into video content with high immersion positive preference (higher preference) and video content with low immersion negative preference (lower preference). , and a horror image showing a negative liking but a peculiar result of high immersion or low immersion, and prompts it.

3. Physiological Measurement Experiment

Based on the factors extracted by the factor extraction experiment, the physiological measurement was performed again using another image of the same subject.

<3.1> Experimental conditions

The subjects were 14 healthy students (age: 19-22, average age: 21.1, 7 males and 7 females) who were studying at the university. The experiment was performed in the same shielded room as in 2.1, and an experimenter was co-located in the same room in order to switch the video content and check the physiological measurement status. In order to realize the adaptation of the body surface temperature to the room temperature, the experiment was carried out after the subjects entered the room for more than 20 minutes. Based on the results of the factor extraction experiment, the video contents of positive and negative preferences were set for each subject. The video content with the highest preference among the immersion sense of 4 or more and the preference of 0.6 or more is regarded as a positive preference (hereinafter, abbreviated as "positive"), and the preference of the immersion sense of 2 or less and the preference of 0.4 or less. The lowest video content is set as a negative preference (hereinafter, abbreviated as "negetive"). Therefore, the contents of the video to be watched are three kinds of "positive", "negative" and "horror".

<3.2> Experimental procedure

It consists of FIG. 13 in the same manner as in the factor extraction experiment. In addition, perceptual change, preference, and immersion evaluation of each video content were performed by VAS, a multiple mood scale (hereinafter abbreviated as MMS), and an immersion survey.

<3·3> Measurement system

The configuration of the measurement system and EEG measurement electrodes is shown in FIG. 15 . The subject wears a continuous sphygmomanometer, electrodes for EEG derivation, and electrodes for ECG derivation on the seat. The subject wore a continuous sphygmomanometer (Finometer model 2, manufactured by Finapres Medical Systems B.V.) at the second joint of the left middle finger, and recorded on the PC at a sampling frequency of 200 Hz. An infrared thermal imaging device (TVS-200X) was installed at an angle capable of measuring the face at a position of 0.7 m in front of the subject. The skin emissivity was set to ε=0.98, the temperature resolution was set to 0.1° C. or less, and the samples were recorded on a PC at a sampling frequency of 1 Hz. EEG was measured by the reference electrode method with the left ear (A1) as the reference. The electrode position for EEG extraction was set to 1 point (Pz) based on the international 10-20 method, and the electrode contact resistance was set to 10 kΩ or less.

Usually, alpha waves are recorded as O1 and O2, but in this study, we focused on the alpha wave energy, which is a stable indicator of brain activity, and specifically calculated its attenuation ratio. The Pz in the vicinity is measured by emphasizing the ease of wearing the electrodes and reducing the measurement pressure on the subject. Electrodes for ECG measurement were worn on the upper sternum (+) and apex (-) according to the NASA lead method in order to minimize the contamination of myoelectric potential. As a ground electrode, the EEG and the ECG were shared as the top of the head (Cz). The EEG signal and the ECG signal were amplified by a digital biological amplifier (5102, manufactured by NFELECTRONIC INSTRUMENTS), and recorded on a PC at a sampling frequency of 200 Hz via a processor box (5101, manufactured by NF ELECTRONIC INSTRUMENTS).

<3.4> Evaluation method

In this study, the correlation between physiological indicators and psychological indicators was analyzed. The physiological indicators are set as mean blood pressure (Mean pressure, hereinafter abbreviated as MP), heart rate (Heart rate, hereinafter abbreviated as HR), stroke volume (StrokeVolume, hereinafter abbreviated as SV), and cardiac output (Cardiac output, hereinafter abbreviated as SV) CO), total peripheral resistance (Total peripheral resistance, hereinafter abbreviated as TPR), nasal skin temperature (Nasal skin temperature, hereinafter abbreviated as NST), electrocardiogram (Electro-cardiograms, hereinafter abbreviated as ECG) and electroencephalogram (Electro -encephalograms, hereinafter abbreviated as EEG). The frequency component of the EEG from 8 Hz to 13 Hz is referred to as the alpha wave, which is known to be pronounced when quiet, with eyes closed, and awake, and attenuated when any condition is broken ⁽⁸⁾ . In this study, Fourier transform (FFT) was performed on 1024 sampling points every 10 seconds for the EEG time series sampled at 200 Hz, and the α-wave power spectrum was obtained every 10 seconds. Furthermore, the average α-wave power in each of the R1 section and the R2 section in FIG. 13 is calculated, and the ratio of the average α-wave power in the R2 section with the average α-wave power in the R1 section as a reference is used as an index for the fluctuation of alertness before and after watching TV. .

NST is known as an indicator of sympathetic nervous system activity that governs peripheral blood flow. The hyperactivity of the sympathetic nervous system is closely related to the temporal variation of inhibition and NST. Therefore, in this study, the variation of NST per 10 seconds was used as a quantitative index of the kinetic energy of the sympathetic nervous system related to video viewing. Positive values indicate inhibition of sympathetic nervous system activity, and negative values indicate hyperactivity of sympathetic nervous system activity. In each increment of the measured nasal thermal image time series, the spatial average temperature in 10×10 pixels of the nose was obtained and set as the NST time series. HF is a high frequency component of 0.15 Hz to 0.4 Hz in the heart rate fluctuation spectrum, and is known as a respiratory sinus arrhythmia component ⁽⁷⁾ .

HF is an index of the parasympathetic nervous system, and increases or decreases according to the hyperactivity or inhibition of the parasympathetic nervous system. The time series of R-wave peak intervals were obtained from ECG samples by thresholding, and after cubic spline interpolation, sampling was performed at 20 Hz. FFT processing is performed on the sampled data every 1 second to obtain the time series of the heart rate fluctuation power spectrum. The number of data processed by FFT is set to 512 points. In the time-series heart rate fluctuation power spectrum, the integral value in the region of 0.15 Hz to 0.4 Hz was obtained, and it was set as the HF time series. Hemodynamic parameters (MP, HR, SV, CO, TPR) are known to show characteristic response modes (Mode I, Mode II) depending on external pressure, which is important in understanding the physiological response of the cardiovascular system to pressure important concept.

Specifically, mode I is characterized by an increase in myocardial contractile activity and an increase in blood volume to skeletal muscle due to vasodilation, which can be said to be an energy-consuming response (active response). On the other hand, mode II is characterized by constriction of peripheral blood vessels and a substantial reduction in heart rate, which can be said to be an energy-saving response (passive coping).

In addition, the psychological indicators are MMS and VAS. MMS targets the temporary mood and emotional state that changes according to the conditions of the subject, through depression and anxiety (hereinafter abbreviated as DA), hostility (hereinafter abbreviated as H), burnout (hereinafter abbreviated as F), active pleasure (Abbreviated as A hereinafter), inactive pleasure (abbreviated as I hereinafter), Affinity (abbreviated as AF hereinafter), concentration (abbreviated as D hereinafter) and astonishment (abbreviated as S hereinafter) consisting of 8 emotional states The scale is indexed ⁽⁹⁾ . Before the start of the experiment and at the end of the experiment, VAS was used to measure 4 kinds of subjective sensations of "awakeness", "pleasure and unpleasantness", "fatigue" and "likeness".

"Pleasure and unhappiness" and "lucidity" were selected as the basic components of Russell's dualism of emotion and were selected as the items of psychological evaluation in this study ⁽¹⁰⁾ . In VAS, a pair of adjectives are arranged at both ends of a line segment, and the examinee selects any position on the line segment, thereby measuring the examinee's psychophysical quantity. For the sentences arranged at both ends of the scale, the alertness is set to "extremely sleepy" - "very alert", the pleasantness and unpleasantness are set to "very unpleasant" - "very pleasant", and the mental level is set to "extremely tired" - "extremely tired"Spirit" and preferences are set to "particularly hate" - "particularly like". Each VAS is prepared on a separate sheet of paper, and the subject is instructed to hand-write in sequence without recursive references. Furthermore, the immersion in the video was measured on a 5-point scale (1: no immersion at all - 5: considerable immersion) after the end of the experiment. In addition, as a statistical analysis method, a corresponding t-test is used for the difference test of each psychological index before and after video viewing, and Wilcoxon's sign order test is used for the amount of change in the entire TEST interval in each physiological index.

<3.5> Results and Research

Statistical analysis was carried out on the psychological indicators, and the psychological responses before and after watching TV images were discussed. In addition, the preference of all the subjects was 0.4 or less for horror image stimulation, so only the immersion was focused. In the 5-stage immersion evaluation after watching horror images, 4 or 5 is classified as horror (focus) (hereinafter abbreviated as "horror (C)"), and 1 or 2 is classified as horror (non-focus) (hereinafter abbreviated as "horror (C)"). (N)”). Afterwards, the results are marked by the classification of "positive", "negative", "horror (C)", "horror (N)". The number of "Terror (C)" is 10, and the number of "Terror (N)" is 4. The averages of all subjects in each emotion scale of MMS are shown in FIGS. 16 to 19 .

N=14 in FIG. 16 , N=14 in FIG. 17 , N=10 in FIG. 18 , and N=4 in FIG. 19 . The averages of all subjects in the "pleasure and unpleasantness", "awakeness", "vigor", "liking", and "immersion" of the immersion are shown in FIGS. 20 to 23 . N=14 in FIG. 20 , N=14 in FIG. 21 , N=10 in FIG. 22 , and N=4 in FIG. 23 .

According to Fig. 16, "Positiveness" was significantly increased in A and AF, and was significantly decreased in F. On the other hand, according to FIG. 17 , the “negativeness” was significantly increased in H and F. This is consistent with the behavior of viewing positive and negative video content separately. "Horror (C)" shows a significant decrease in A and I in FIG. 18 , and it can be presumed that it generally evokes unpleasant emotions. In addition, compared with FIG. 19 , there are many scales with significant differences in FIG. 18 .

From this, it can be inferred that, with regard to terror, the change in emotion is greater when immersed than when not immersed. According to Figure 20, "Positivity" significantly increased in pleasure and unhappiness, wakefulness, and vitality. From this, it can be considered that the video content that is preferred brings positive comfort to the subject along with "refreshing feeling" and "comfortable feeling".

On the contrary, according to FIG. 21 , since “negativeness” was significantly reduced in pleasantness and unpleasantness, alertness, and vitality, it can be considered that the video content of negative liking brought “depression” and “unpleasantness” to the subject. of negative unhappiness. In addition, according to FIG. 22 , since “fear (C)” was significantly increased in the sense of sobriety, and significantly decreased in the sense of pleasure, unpleasantness, and vitality, it is considered that “a sense of wakefulness” and “a sense of unpleasantness” accompanied the Positive unpleasantness.

Next, the physiological responses during TV video viewing and before and after TV video viewing are discussed. Fig. 24 shows the average temporal variation among subjects of each physiological index. In FIG. 24, N=14. From the top are NST, alpha wave, HF, MP, HR, SV, CO, TPR. Set the start of the TEST interval to 0. Error bars in the figure represent standard errors per 10s. In addition, the baselines of NST, MP, HR, SV, CO, TPR, and the R1 section of immersion were set to 0, and the baseline of the R1 section of alpha waves and HF was set to 1 for normalization. The significant probability p of the displacement of the baseline related to the Wilcoxon's sign order test is shown in Figure 25 (+: p<0.1, *: p<0.05, **: p<0.01). P in the table represents a positive response to each indicator, and N represents a negative response. In Fig. 25, N=14.

According to FIG. 24 , when the time-series changes of the respective indexes are observed, both the “positiveness” and the “negativeness” of the NST decrease with the start of the TEST interval, so it can be presumed that the sympathetic nerve is hyperactive. No significant changes were seen in alpha waves. According to Figure 25, HF did not see a significant change in "Positive" and decreased significantly in "Negative". HF is an index of parasympathetic nervous system activity, and it increases or decreases according to the activation or inhibition of parasympathetic nerves, so it can be considered that parasympathetic nerves are "negatively" inhibited. This is consistent with the explanation in the NST above.

Fig. 26 shows the average temporal variation among subjects of each physiological index of the horror video. In FIG. 26, N=14. According to FIG. 26, since the NST of "fear (N)" decreased, it is presumed that the sympathetic nerve was hyperactive. None of the significant results were obtained for alpha waves. In HF, it was found that "fear (C)" was significantly decreased, and "fear (N)" was significantly increased. For HF, in "Terror (N)" is inconsistent with the interpretation in the NST above. Considering this is due to a difference in mechanism. As shown below, "Terror (N)" shows a proactive response characterized by an increase in TPR. As a result, it is considered that the blood flow in the peripheral blood vessel portion decreases and the NST decreases.

Next, according to the patterns of immersion and preference, "positive", "negative", "horror (C)" and "horror (N)" were classified to compare the hemodynamic responses. As a result, as shown in FIG. 27 , the stress response method and the axis of preference can be summarized. In Fig. 27, N=14.

Both "positive" and "horror (C)" have higher immersion, but "positive" is a positive preference, and horror (C) is a negative preference. However, both parties saw an increase in MP, HR, SV, and CO, and a decrease in TPR. This is a typical mode I response (active response), which is dominated by an increase in myocardial contractile activity.

That is, it is clear that when actively responding, regardless of preference, one is immersed in TV video content. In addition, by analogy with the results of the factor extraction experiment, it can be expected that in the physiological measurement experiment, the subjects who have a high preference for horror images also have a high immersion sense, and are located in the upper right area of FIG. 27 , but there is no match. On the other hand, "Positivity" and "Terror (N)" have lower immersion and liking. However, their physiological responses are different. That is, in "negative", the decrease in TPR and HR was observed, but the change in MP was not observed. It can be considered that it is not either active response or passive response, but no response. In addition, in FIG. 27, N=14.

On the other hand, in "Terror (N)", although there is no significant change in HR, it is characteristic that TPR and MP increase significantly. This is a typical pattern II response (passive coping), which is dominated by an increase in MP caused by an increase in peripheral vasoconstriction. From this, it is clear that when stress coping is not shown, the preference is generally low and the TV video content is not immersed, but the related horror video shows passive coping. That is, it is obvious that the viewing state cannot be determined based on the preference for TV video alone, and the viewing state can be classified according to the stress response method.

4. Summary

In this study, physiological and psychological measures of viewing video content with different preferences were implemented, and an attempt was made to classify viewing styles according to stress coping styles with physiological evidence. Measurement of hemodynamic parameters (MP, HR, SV, CO, TPR) as indicators of cardiovascular system, measurement of alpha wave power spectrum of EEG as indicators of central nervous system, measurement of nasal skin temperature and heart rate fluctuation HF components as autonomic nerves It also includes the simultaneous measurement of video preferences and psychological questionnaires. Statistical analysis related to physiological and psychological states is carried out to quantitatively evaluate the physiological and psychological effects of viewing video content.

As a result, the classification of psychological indicators related to preferences in this study has not been evaluated at all in existing research, so by combining stress responses and preferences in this study, a new classification of TV viewing can be obtained. As a conclusion, when actively coping, the TV image is "positive" and "horrible (C)", when passively coping, the TV image is "horrible (N)", and when there is no stress coping, the TV image is "negative" ". That is, it is obvious that the viewing mode cannot be determined only by the preference for TV images, but the viewing mode can be determined based on the pressure response of the hemodynamic parameters.

<References>

(1) Hirata Akihiro, Morto Eimi, Aramaki Yang: "The Present of TV Viewing and Media Utilization (2) ~ From the "Japanese and TV 2010" Survey ~", Screening Research and Survey, pp.2-27 (2010 )

(2) Aramaki Yang, Masuda Tomoko, Nakano Sachiko: "How TV will face people in their 20s", Screening Research and Survey, pp.2-21 (2008)

(3) Y. Siki, Y. Murakami and Y. Huzita: "Television viewing andsimultaneous use of new media: anethnographic study of young people in Japan" Institute for Media and Communication Research Keio University, Vol.59, No.1, pp .131-140 (2009) (in Japanese) Shiki Yuko, Murayama Yo, Fujita Yuko: "Youth's TV Viewing and Media Utilization Behavior: An Ethnographic Survey of Viewers from College Students" Journal of Keio University's Institute of Media Communication , Vol.59, No.1, pp.131-140 (2009)

(4) S.Nomura, Y.Kurosawa, N.Ogawa, C.M.Althaff Irfan, K.Yajima, S.Handri, T.Yamagishi, K.T.Nakahira, and Y.Fukumura: "Psysiological Evaluation of astudent in E-learning Sessions by "Hemodynamic Response", IEEJ Trans.EIS, Vol.131, No.1, pp.146-151 (2011) (in Japanese) Nomura Sosaku, Irfan C.M.Althaff, Yamagishi Takao, Kurosawa, Yajima Kuniaki, Naka Hira Katsuko, Ogawa Nobuyuki, HANDRI Santoso, Fukumura Yumi: "Study on the Physiological Evaluation of E-Learning Attendees through Hemodynamic Parameters", Theory of Electricity C, Vol.131, No.1, pp.146-151 (2011 )

(5) T. Watanuki and A. Nozawa: "Visualization of a feeling of concentration for TV contents", Bulletin of Institute of Electrical and Electronic Engineers of measurement, Vol.IM-12, No.63, pp.19-25 (2012) (inJapanese) Takuya Mianuki and Akio Nozawa: "Visualization of Immersion in TV Content", Institute of Electrical and Electronics Engineering Research Fund, Vol.IM-12, No.63, pp.19-25 (2012)

(6) K.Mori, C.Ono, Y.Motomura, H.Asoh, and A.Sakurai: "E mpirical Analysisof User Preference Models for Movie Recomme ndation", IEIC TechnicalReport.NC.neurocomputing, Vol.104, No. 759, pp.77-82 (2005) (in Japanese) Kurokawa Shigeru, Ono Tomohiro, Honmura Yoichi, Aso Hideki, Sakurai Akio: "Experimental Evaluation of User Preference Models for Movie Content Recommendation", letter Journal of the Society. NC. Neural Computation, Vol.104, No.759, pp.77-82 (2005)

(7) Yukio Sawada: "Hemodynamic Responses; New Physiological Psychology Volume I (edited by Kiyoshi Fujisawa, Tomoji Kakigi, Katsuo Yamazaki)", Kitaoji Study, Chapter 10, p.187 (1998)

(8) J.A.Russell: "A circumplex model of affect", J.Personality and Social Psychology, Vol.39, pp.1161-1178 (1980)

<Experimental example 3>

1. The purpose of the experiment

In recent years, the digital and multi-channelization of television projections has been continuously promoted. However, with the spread of miniaturized and high-performance information communication equipment and the expansion of the information network environment brought about by the development of IT technology, the phenomenon of de-television as entertainment has accelerated, and the reason for watching television has become diversification. Hirata et al. cited "in order to understand the events and dynamics in society", "in order to relieve fatigue or relaxation", and "in order to deepen or expand interpersonal relationships" as the reasons for watching TV ⁽¹⁾ . In addition, with the de-television of the reasons for watching TV and the way of watching, there is a change in the viewpoint of weak participation such as a less concerned attitude towards watching or a weakening of watching habits. Shiki et al. exemplified "dedicated viewing", which focuses on watching TV, "simultaneous viewing", which is parallel viewing with other life activities such as housework, eating, and learning, and "don't care about viewing", which involves watching many programs at the same time when changing channels. How to watch. In this way, it is clear that there are various ways to watch, and they change according to factors such as time, place, mood, etc.

However, most of these studies that categorize viewing patterns have used psychological responses such as public opinion surveys and post-event introspective evaluations, and there have been no studies that have used physiological responses to categorize viewing patterns. Since the images and sounds of TV are visual and auditory stimuli to the living body, they can be regarded as stressors themselves. Stressors prompt the organism to respond to stress, which brings about physiological and psychological changes in response to stress.

When various contents of television are regarded as stressors, it can be expected that the response to the contents, that is, the way of viewing, varies in response to physiological stress. Nomura et al. classified the corresponding status of e-learning content by cardiovascular system indicators ⁽⁴⁾ .

Therefore, so far, attempts have been made to classify TV video content viewing styles and preferences based on cardiovascular system indicators ⁽⁵⁾ . In addition, not only classification but also construction of an estimation model is studied.

Kurokawa et al. constructed a user preference model for video content by using Naive Bayes, decision tree, Bayesian network, and neural network, and extracted the prediction accuracy and important variables for content evaluation ⁽⁶⁾ . However, this study also used only a model of psychological response, and did not conduct a study to construct an estimation model of preferences and viewing patterns using physiological responses.

The results of the studies so far suggest the possibility of classifying the physiological and psychological states of television viewers, such as content preference and immersion, based on the stress coping style and heart rate based on hemodynamic parameters. Therefore, the purpose of this study is to conduct an experimental study on the estimation of the physiological and psychological state of the viewers related to the above-mentioned television viewing based on the cardiovascular system index. Feature vectors are extracted from cardiovascular system indices, and various estimation models for preference, viewing style, and excitement-sedation of TV video content are created and evaluated.

2. Experiment

Cardiovascular system measurement using a continuous sphygmomanometer during viewing of TV video content was performed. Then, based on the hemodynamic parameters, pattern recognition using a hierarchical neural network was performed to estimate preference, viewing style, and excitement-sedation when viewing TV video content.

<2.1> Experimental procedure

As shown in FIG. 28 , this experiment consisted of 5 minutes of video viewing and 1 minute of quiet eye-closed states R1 and R2 before and after it. The preference and immersion of various video contents were evaluated by the Visual Analogue Scale (hereinafter abbreviated as VAS) before and after viewing the video, and the immersion survey before and after viewing. In addition, changes in subjective excitement-sedation felt by the subjects were recorded in real time during the viewing of the video.

<2.2> Experimental conditions

The subjects were 14 healthy students (age: 19-22, average age: 21.4, 7 males and 7 females) currently studying at Japanese universities. The subjects were fully explained to the subjects orally and in writing in advance about the content of the experiment, the purpose, and the subjects of the investigation, and their consent to the experimental assistance was confirmed by signature. The measurement was performed in a shielded room with no convection at a room temperature of 26.0±1.6°C, and an experimenter was co-located in the same room in order to switch the video content and confirm the physiological measurement status. In order to realize the adaptation of the body surface temperature to the room temperature, the experiment was carried out after the subjects entered the room for more than 20 minutes.

The subject used an LCD TV (55-inch, HX850, BRAVIA, SONY) installed at a position 2m in front of the seat, and watched video content with positive preferences and high immersion (hereinafter abbreviated as "positive"), negative preferences, There are three types of video content with low immersion (hereinafter, abbreviated as "negative") and horror video showing the peculiar result that there is a large individual difference in immersion despite low preference. The video content is played by a DVD player (SCPH-3900, SONY).

<2·3> Measurement system

The measurement system is shown in FIG. 29 . The subject was wearing a continuous sphygmomanometer in the seat. The subject wore a continuous sphygmomanometer (Finometer model 2, manufactured by Finapres Medical Systems B.V.) at the second joint of the left middle finger, and recorded on the PC at a sampling frequency of 200 Hz. In addition, a keyboard (K270, manufactured by Logicool) was installed in front of the keyboard, and the excitation and sedation were sequentially and relatively recorded using software for inputting the subjective excitation-sedation changes felt by the examinee in real time through the up and down keys of the keyboard.

<2.4> Evaluation method

The physiological indicators are set as mean blood pressure (Mean pressure, hereinafter abbreviated as MP), heart rate (Heart rate, hereinafter abbreviated as HR), stroke volume (Stroke Volume, hereinafter abbreviated as SV), and cardiac output (Cardiac output, hereinafter). Abbreviated as CO), total peripheral resistance (Total peripheral resistance, hereinafter abbreviated as TPR). Hemodynamic parameters (MP, HR, SV, CO, TPR) are known to show characteristic response modes (Mode I, Mode II) according to external pressure and are important in understanding the physiological response of the cardiovascular system to pressure the concept of. Specifically, mode I is characterized by an increase in myocardial contractile activity and an increase in blood volume to skeletal muscle due to vasodilation, which can be said to be an energy-consuming response (active response). On the other hand, mode II is characterized by constriction of peripheral blood vessels and a substantial reduction in heart rate, which can be said to be an energy-saving response (passive coping) ⁽⁷⁾ . The baseline of the R1 interval was set to 0 to normalize MP, HR, SV, CO, TPR.

In addition, as a psychological index, VAS was measured using four subjective sensory quantities of "awakeness", "pleasure and unpleasantness", "vitality", and "liking" before the start of the experiment and at the end of the experiment. "Pleasure and unhappiness" and "lucidity" were selected as the basic components of Russell's dualism of emotion and were selected as the items of psychological evaluation in this study ⁽⁸⁾ . In VAS, a pair of adjectives are arranged at both ends of a line segment, and the examinee selects any position on the line segment, thereby measuring the examinee's psychophysical quantity. For the sentences arranged on both ends of the scale, the alertness is set to "extremely sleepy" - "very alert", the pleasantness and unpleasantness are set to "very unpleasant" - "very pleasant", and the vitality is set to "extremely tired" - "extremely energetic" , and preferences are set to "particularly hate" - "particularly like".

Each VAS is prepared on a separate sheet of paper, and the subject is instructed to hand-write in sequence without recursive references. Furthermore, the immersion in the video was measured on a 5-point scale (1: no immersion at all - 5: considerable immersion) after the end of the experiment. In addition, a corresponding t-test is used for the test of the difference before and after viewing the video of each psychological index, and Wilcoxon's sign order test is used for the amount of change in the entire TEST interval in each physiological index. Excitation-sedation was normalized to the summed maximum.

Pattern recognition based on a hierarchical neural network was used for generation of preference and viewing mode classification models, derivation of discrimination rates, and generation of arousal-sedation estimation models for each subject, and derivation of estimated values.

The structure of the hierarchical neural network is shown in FIG. 30 . The learning rule is set as the error back propagation algorithm, and the output function is set as the Sigmoid function. The layers are input layer, intermediate layer, and output layer. The cross-validation method is used to estimate the accuracy of the model. In addition, as shown in FIG. 31 , a determination time t _d is assumed for the time-dependent variation of each index. t _d is 300s in video viewing. For each model, in each feature quantity time series in the t _d interval, focusing on the slope of each Δt _i , a series of slopes at every 10 s interval is set as the feature vector S. The number of elements of S is 30. Matching this, in the neural network of this paper, the number of units in the input layer is set to 30, and the number of units in the middle layer is set to 16.

<2.4.1> Favorite Classification Model

The study was conducted using the heart rate as a feature quantity and Δt _i as 10s, 20s, 30s, 40s, 50s, and 60s. The total learning data is a total of 28 patterns for each preference (positive, negative) of the 14 subjects, of which 27 patterns are used as learning data, and the remaining 1 pattern is used as unknown data to evaluate the accuracy of the preference classification model. After learning the feature vector S for "positive" and "negative", the accuracy of the preference classification model is evaluated by inputting unknown data. The output layer is 2 units of "Positive" and "Negative".

<2.4.2> Viewing Mode Classification Model

A hemodynamic parameter was used as a characteristic quantity, and Δt _i was set to 10s, 20s, 30s, 40s, 50s, and 60s and studied. The total learning data is a total of 42 modes of each stress coping style (active coping, passive coping, and stress-free coping) of the 14 subjects. Among them, 41 modes are used as learning data, and the remaining 1 mode is used as unknown data to evaluate preference classification accuracy of the model. The output layer is three units of active response, passive response, and stress-free response.

<2.4.3> Excitement-sedation presumption model

A hemodynamic parameter was used as a characteristic quantity, and Δti was set to 30 s. The total learning data is a total of 28 patterns of excitation-sedation values in each preference (positive, negative) of the 14 subjects, of which 27 patterns are used as learning data, and the remaining 1 pattern is used as unknown data. Likes the accuracy of the classification model. The output layer is 1 unit of excitation-sedation value.

<2.5> Results and Research

In the VAS, the difference between before and after viewing the video averaged for all subjects in each emotion scale is shown in Fig. 32 (+: p<0.1, *: p<0.05, **: p<0.01) out. In FIG. 32, N=14. According to FIG. 32 , all the subjects had low preference for horror image stimulation. Therefore, in the five-stage immersion evaluation after viewing the horror video, 4 or 5 is classified as horror (concentration) (hereinafter abbreviated as "horror (C)"), and 1 or 2 is classified as horror (unfocused) (hereinafter abbreviated as "fear (C)"). for "horror (N)"). The results after that were classified as "positive", "negative", "horror (C)", "horror (N)". In addition, the number of "Terror (C)" is 10, and the number of "Terror (N)" is 4.

The significance probability p of Wilcoxon's sign order test related to the shift of the baseline from the overall TEST interval of MP, HR, SV, CO, TPR is shown in Figure 33 (+: p<0.1, *: p<0.05, * *: p<0.01). P in the table represents a positive response to each indicator, and N represents a negative response. In FIG. 33, N=14.

According to FIG. 33 , the classification of the viewing methods when viewing TV video images is performed. Both "positive" and "horror (C)" have high immersion, but "positive" is a positive preference, and "horror (C)" is a negative preference.

However, both parties saw an increase in MP, HR, SV, and CO, and a decrease in TPR. This is a typical mode I response (active response), which is dominated by an increase in myocardial contractile activity. That is, it is clear that when actively responding, regardless of preference, one is immersed in TV video content.

On the other hand, "Negativeness" and "Terror (N)" have lower immersion and liking. However, their physiological responses are different. That is, in "negative", the decrease in TPR and HR was observed, but the change in MP was not observed. It is considered that it is not either active response or passive response, but no response. On the other hand, although there was no significant change in HR of "horror (N)", significant increases in TPR and MP were characteristic. This is a typical pattern II response (passive coping), which is dominated by an increase in MP caused by an increase in peripheral vasoconstriction.

From this, it is clear that when stress response is not shown, the preference is generally low and the TV video content is not immersed, but the related horror video shows passive response. That is, it is obvious that the viewing state cannot be determined only by the preference for TV video, and the viewing state can be classified according to the stress response method. Furthermore, in Fig. 33, HR significantly increased in "Positive" and significantly decreased in "Negative".

34 and 35 show the results of the discrimination rate when judging preference and viewing mode based on the estimation model constructed using the neural network. In FIGS. 34 and 35 , when Δt _i is 50s, the positive discrimination rate of preference is 83.3%, and the positive discrimination rate of viewing mode is 75%, which are higher than other Δt _i .

In addition, the measured excitation-sedation and the predicted excitation-sedation when the excitation-sedation was estimated are shown in FIG. 36 .

FIG. 36 shows the results of subject A with the least error between the measured excitation-sedation and the predicted excitation-sedation. In "Positive" on Figure 36, as the measured excitation-sedation increases, the predicted excitation-sedation also increases.

Furthermore, in "Negative" under Figure 36, as the measured excitation-sedation decreases, the predicted excitation-sedation also decreases. Next, Fig. 37 shows the average error of each subject when the absolute value of the difference between the measured excitation-sedation and the predicted excitation-sedation is used as the error. In Fig. 37 , the average error of "positive" is 0.10 to 0.37, the average error of "negative" is 0.11 to 0.29, and the average error among all subjects is 0.17. That is, excitation-sedation is predicted with an average error of about 17%. Also, a graph showing a series of measured excitation-sedation and predicted excitation-sedation relationships obtained every 10 s interval for all subjects is shown in FIG. 38 . In Fig. 38, the correlation coefficient is 0.89, which gives a high correlation. In addition, as can be seen from FIG. 38 , the distributions of the measured excitation-sedation and the predicted excitation-sedation are substantially uniform regardless of the magnitude of the value.

3. Summary

In this study, we extracted feature vectors from cardiovascular system indexes for the purpose of estimating viewing patterns, preferences, and excitement-sedation for TV video content using neural networks. and generation and evaluation of putative models of excitation-sedation.

As a result, "Positiveness" of positive liking and high immersion and "horror (C)" of low liking and high immersion were classified as active coping, and "horror (N)" of negative liking and high immersion was classified as passive coping, and negative liking was classified as passive coping. , Low immersion "negative" is classified as stress-free coping. Furthermore, the "positive" and "negative" heart rates are significantly different, and results are obtained that follow the general characteristics of the heart rate response. In addition, when using the estimation model to discriminate preference and viewing method, the highest positive discrimination rate was 83.3% for preference and 75% for viewing method.

As a result of the estimated excitation-sedation for each subject, the average error between the measured excitation-sedation and the predicted excitation-sedation was 10-37% in "positive" and 11-29% in "negative", The average among all subjects was 17%. Furthermore, it can be seen that the correlation coefficient between the measured excitation-sedation and the predicted excitation-sedation is 0.89, which is a strong positive correlation. As described above, this suggests the possibility of estimating preference, viewing style, and excitement-sedation when viewing television video content from hemodynamic parameters and heart rate. In the future, we plan to further improve the accuracy through comparison and research with other physiological indicators and other discriminators.

<References>

(1) Tsuyoshi Nishigaki: "Will TV Resurrect Through the Internet?" Tsukijian Minfang (2001)

(2) Fujiwara Kotatsu and Saito Yumiko: "Public Opinion Survey Report's Evaluation and Expectation of Television - From the "Report on the Role of Television" Survey Report-", NHK Screening Research and Survey June Issue, pp.2-15 (1989)

(3) Hiroaki Ohnogi: "Chapter 9 The Attitudes of Female College Students to Watching TV and Factors 2 and 3: From a preliminary survey related to the viewer's personality, concern for the content of the program, and the purpose of watching TV (Program Analysis and Watching Learning) Research on Behavior: Towards the Development of a Classification System for Screening Educational Programs), Research Report, Vol.18, pp.153-172 (1990)

(4) Y.Takahashi and S.John: "Recommendation Models of TelevisionProgram Genre, Based on Survey and Analysis of Beha viour in WatchingTelevision: Toward Human Content Interface Design (2)", Bulletin of Japanese Society for Science of De sign, Vol. 46, No.133, pp.71-80 (1999) (in Japanese) Takahashi Yasushi, Shackleton John: "Program Type Recommendation Model Based on Investigation and Analysis of TV Watching Behavior: Interface Design for Humanized Content (2)", Design Studies, Vol.46, No.133, pp.71-80 (1999)

(5) Yumiko Tomomune and Yumiko Hara: "TV as a "Relaxing Time Device"-The Relationship between Viewing Attitude and the Comprehensive Variety of Programs-", Screening Research and Survey, Vol.51, pp.2-17 (2001)

(6) S.Nomura, Y. Kurosawa, N.Ogawa, C.M.Althaff Irfan, K.Yajima, S.Handri, T.Yamagishi, K.T.Nakahira, and Y.Fukumura: “Psysiological Evaluation of astudent in E-learning Sessions by "Hemodynamic Response", IEEJ Trans.EIS, Vol.131, No.1, pp.146-151 (2011) Nomura Sosaku, Irfan C.M.Althaff, Yamagishi Takao, Kurosawa, Yashima Kuniaki, Nakahira Katsuko, Nobuyuki Ogawa, HANDRI Santoso, Yumi Fukumura: "Study on the Physiological Evaluation of E-Learning Attendees by Hemodynamic Parameters", Theory of Electricity C, Vol.131, No.1, pp.146-151 (2011)

(7) Yukizho Sawada: "Hemodynamic Responses; New Physiological Psychology Volume I (edited by Kiyoshi Fujisawa, Shinji Kakigi, Katsuo Yamazaki)", Kitaoji Study, Chapter 10, pp.187 (1998)

(8) A.Michimori: "Relationship between the alpha attenu ationtest, subjective sleepiness and performance test", 10th Symposium on Human Interface, Vol.10, No.1413, pp.233-236 (1992)

(9) M. Terasaki, Y. Kishimoto, and A. Koga: "Construction of a multiple moodscale", The Japanese journal of psychology, Vol.62, No.6, pp.350-356 (1992) Terasaki Shoji, Yoichi Kishimoto and Aito Koga: "The Generation of the State Scale of Multifaceted Senses", Journal of Psychological Research Papers, Vol.62, No.6, pp.350-356 (1992)

(10) J.A.Russell: "A circumplex model of affect", J.Personality and Social Psychology, Vol.39, pp.1161-1178 (1980)

Industrial Applicability

The stress response method determination system of the present invention can analyze the nature of the stress of the subject in a non-contact state, so it can be used as a means for grasping the stress state of a worker working in a factory or the like, and for grasping the driving of a car. Mechanisms for the stress state of the driver during the course, means for grasping the stress state of the students attending the lecture, and the like are used in a wide range of technical fields.

Description of reference numerals

100 Stress coping style determination system

110 Biometric information acquisition device (biological information acquisition unit)

120 Judging device (judging part)

121 Feature storage unit for judgment

122 Specific site reaction detection unit

123 Response mode determination section

130 Learning Devices (Machine Learning Division)

131 Learning data storage unit

132 Feature Extraction Section

133 Feature Quantity Learning Department

134 Learning the model

P subject

IF face image

S1 Determination feature storage processing (determination feature storage step)

S2 Specific site reaction detection processing (specific site reaction detection step)

S3 Response mode determination processing (response mode determination step)

S11 Learning data storage processing (learning data storage step)

S12 Feature extraction processing (feature extraction step)

S13 Feature learning process (feature learning step)

S21 Clustering processing (clustering step)

S22 Image extraction processing (image extraction step)

S23 Edge extraction processing (edge extraction step)

S24 Fractal analysis processing (fractal analysis step).

Claims (21)

1. A stress coping mode determination system, characterized in that it has: The biometric information acquisition unit acquires the subject's biometric information in a non-contact state; a determination unit for determining a stress coping style of the subject based on the biological information and a predetermined response pattern, The response mode is determined from hemodynamic parameters. 2. The stress coping mode determination system according to claim 1, characterized in that: The hemodynamic parameters include multiple parameters of mean blood pressure, heart rate, cardiac output, stroke volume, and total peripheral vascular resistance. 3. The stress coping mode determination system according to claim 1 or 2, characterized in that: The biological information is a face image. 4. The stress coping mode determination system according to claim 3, characterized in that: The facial image is a thermal facial image or a visible facial image. 5. The stress coping mode determination system according to claim 3 or 4, characterized in that: The determination unit determines the stress coping style of the subject by observing the stress response of a specific part of the face included in the face image. 6. The stress coping mode determination system according to claim 5, characterized in that: The response mode includes three modes of "active response", "passive response" and "no response". 7. The stress coping mode determination system according to claim 6, characterized in that: The determining unit has a feature amount storage unit for determination that stores a spatial feature amount corresponding to "active response", a spatial feature amount corresponding to "passive response", and a spatial feature amount corresponding to "no response", Based on the biometric information and each spatial feature value stored in the feature value storage unit for determination, it is determined whether the stress response method shows which of "active response", "passive response" and "no response" a pattern of ways. 8. The stress coping mode determination system according to claim 7, characterized in that: The feature value stored in the feature value storage unit for determination is the feature value extracted by the machine learning unit, The machine learning section has: The data storage unit for learning stores a plurality of facial images for learning with labels corresponding to "active response", "passive response" and "no response" respectively; A feature extraction unit that uses the learned model to extract the spatial feature of the face image from the face image for learning; A feature value learning unit, based on the relationship between the extraction result obtained by the feature value extraction unit and the label attached to the learning face image as the extraction target, changes the network parameters of the learned model so that the The extraction accuracy of the spatial feature obtained by the feature extraction unit increases. 9. The stress coping mode determination system according to claim 7 or 8, characterized in that: The spatial feature amount is a fractal dimension calculated based on the face image of the subject. 10. A program for causing a computer to function as a mechanism for determining a stress coping style of a subject, comprising: The feature storage step for determination is to store the spatial feature corresponding to "active response", the spatial feature corresponding to "passive response", and the spatial feature corresponding to "no response"; The determining step is to determine whether the stress coping style of the subject is "active coping", "passive coping" and "no response" based on the facial image of the subject and each spatial feature quantity stored in the above-mentioned determining feature quantity storing step. Which of the response modes in "Coping", The response pattern is determined by hemodynamic parameters. 11. The program of claim 9, having: The data storage step for learning is to store a plurality of facial images for learning with labels corresponding to "active response", "passive response" and "no response" respectively; Feature extraction step, using the learning completed model to extract the spatial feature of the facial image for learning; A learning step, based on the relationship between the extraction result obtained by the feature amount extraction step and the label labeled with the learning facial image as its extraction object, changing the network parameters of the learned model so that the feature amount is determined by the The extraction accuracy of the feature quantity obtained in the extraction step becomes higher, The feature value storage step for determination is a step of storing the spatial feature value extracted by the feature value extraction step. 12. The program of claim 10 or 11, wherein The spatial feature amount is a fractal dimension calculated based on the face image of the subject. 13. A method for determining a stress coping style, characterized in that it has: The biological information acquisition step is to acquire the biological information of the subject in a non-contact state; The determining step is to determine the stress coping mode of the subject based on the biological information and the predetermined response mode, The response pattern is determined by hemodynamic parameters. 14. A learning device, characterized in that it has: a data storage unit for learning, storing a plurality of facial images for learning with labels corresponding to the response patterns determined by the hemodynamic parameters; The feature extraction part uses the learning completed model to extract the spatial feature of the face image of the examinee from the face image for learning; A feature value learning unit, based on the relationship between the extraction result obtained by the feature value extraction unit and the label attached to the learning face image as the extraction target, changes the network parameters of the learned model so that the The extraction accuracy of the spatial feature obtained by the feature extraction unit increases. 15. The learning device of claim 14, wherein The spatial feature amount is a fractal dimension calculated based on the face image of the subject. 16. A learning method, characterized in that it has: The data storage step for learning is to store a plurality of facial images for learning with labels corresponding to the response patterns determined by the hemodynamic parameters; The feature extraction step, using the learned model to extract the spatial feature of the subject's facial image from the facial image for learning; The feature amount learning step is to change the network parameters of the learned model so that the model is The extraction accuracy of the spatial feature obtained in the feature extraction step becomes higher. 17. The learning method of claim 16, wherein The spatial feature amount is a fractal dimension calculated based on the face image of the subject. 18. A program for causing a computer to function as a mechanism for learning a spatial feature amount of a face image of a subject, comprising: The data storage step for learning is to store a plurality of facial images for learning with labels corresponding to the response patterns determined by the hemodynamic parameters; The feature extraction step, using the learning completed model to extract the spatial feature of the subject's facial image from the facial image for learning; The feature value learning step is to change the network parameters of the learned model so that the model is The extraction accuracy of the spatial feature obtained in the feature extraction step becomes higher. 19. The program of claim 18, wherein The spatial feature amount is a fractal dimension calculated based on the face image of the subject. 20. A learned model, characterized in that, The learned model is generated by performing machine learning on the spatial feature amount of the subject's face image by using a plurality of face images for learning labeled corresponding to the response patterns determined by the hemodynamic parameters as training data. 21. The learned model of claim 20, wherein The spatial feature amount is a fractal dimension calculated based on the face image of the subject.

Images