JP7537207B2 - Information processing device, information processing method, and program - Google Patents

Description

The present invention relates to an information processing device, an information processing method, and a program.

Technology is known that uses a wearable device worn by a user to detect the user's movements and identify the user's behavior.

For example, Patent Document 1 describes a mobile phone device that detects acceleration information of a user and controls an operation mode using the detected acceleration information. Patent Document 2 describes a motion information measurement system that measures motion information of a desired part of a user's body and recognizes the motion state of the entire body.

JP 2003-46630 A JP 2004-184351 A

Here, it is required to identify the user's behavioral pattern based on information about the user's behavioral state.

The present invention aims to provide an information processing device, an information processing method, and a program that can identify a user's behavioral pattern based on information related to the user's behavioral state.

An information processing device according to one aspect of the present invention includes a behavioral state sensor that detects behavioral state information related to a user's behavioral state, a biosensor that detects bioinformation related to the user's bioinformation, an autonomic nerve activity calculation unit that calculates the autonomic nerve activity of the user based on the bioinformation, and an output control unit that changes the intensity of the output from the output unit according to the intensity of the autonomic nerve activity.

An information processing method according to one aspect of the present invention includes the steps of detecting behavioral state information related to a behavioral state of a user, detecting bioinformation related to the bioinformation of the user, calculating the autonomic nerve activity of the user based on the bioinformation, and changing the intensity of the output from an output unit according to the intensity of the autonomic nerve activity.

A program according to one aspect of the present invention causes a computer to execute the steps of detecting behavioral state information related to a user's behavioral state, detecting bioinformation related to the user's bioinformation, calculating the autonomic nerve activity of the user based on the bioinformation, and changing the intensity of the output from an output unit according to the intensity of the autonomic nerve activity.

According to the present invention, it is possible to identify a user's behavioral pattern based on information about the user's behavioral state.

FIG. 1 is a schematic diagram illustrating an information processing apparatus according to the first embodiment. FIG. 2 is a block diagram showing an example of the configuration of the information processing device according to the first embodiment. FIG. 3 is a flowchart showing an example of a processing flow of the information processing device according to the first embodiment. FIG. 4 is a diagram for explaining a multidimensional space in which behavior pattern information is generated. FIG. 5 is a diagram for explaining a format for storing behavior patterns. FIG. 6 is a flowchart showing an example of a processing flow of the information processing device according to the second embodiment. FIG. 7 is a diagram for explaining daily behavior and non-daily behavior. FIG. 8 is a block diagram showing an example of the configuration of an information processing device according to the third embodiment. FIG. 9 is a flowchart showing an example of a processing flow of the information processing device according to the third embodiment. FIG. 10 is a graph showing an example of a pulse wave. FIG. 11 is a diagram for explaining a format for storing behavior patterns. FIG. 12 is a block diagram showing an example of the configuration of an information processing device according to the fourth embodiment. FIG. 13 is a diagram for explaining an example of the correction data. FIG. 14 is a flowchart showing an example of the flow of processing by the information processing device according to the fourth embodiment. FIG. 15 is a diagram for explaining an example of the correction data. FIG. 16 is a flowchart showing an example of the flow of processing by the information processing device according to the fifth embodiment. FIG. 17 is a block diagram showing an example of the configuration of an information processing device according to the sixth embodiment. FIG. 18A is a diagram illustrating an example of user history data. FIG. 18B is a diagram illustrating an example of user history data. FIG. 18C is a diagram illustrating an example of user history data. FIG. 19 is a flowchart showing an example of the flow of the learning process according to the sixth embodiment. FIG. 20 is a flowchart showing an example of the flow of processing by the information processing device according to the sixth embodiment. FIG. 21 is a diagram illustrating an example of the configuration of an information processing system according to the seventh embodiment. FIG. 22 is a block diagram illustrating an example of the configuration of a server device according to the seventh embodiment. FIG. 23 is a flowchart showing an example of a process flow of a server device according to the seventh embodiment. FIG. 24 is a flowchart showing an example of the flow of the learning process according to the seventh embodiment. FIG. 25 is a flowchart showing an example of the flow of processing by the information processing device according to the seventh embodiment. FIG. 26 is a block diagram showing an example of the configuration of an information processing device according to the eighth embodiment. FIG. 27A is a diagram for explaining a method for relatively displaying the transition of activity scores over time. FIG. 27B is a diagram for explaining a method for relatively displaying the transition of activity scores over time. FIG. 27C is a diagram for explaining a method for relatively displaying the transition of activity scores over time. FIG. 27D is a diagram for explaining a method of relatively displaying the transition of activity scores over time. FIG. 27E is a diagram for explaining a method for relatively displaying the transition of activity scores over time. FIG. 28 is a diagram for explaining a method for outputting information indicating a change over time in activity scores.

Hereinafter, an embodiment of the present invention will be described in detail with reference to the attached drawings. Note that the present invention is not limited to this embodiment, and when there are multiple embodiments, it also includes configurations in which the respective embodiments are combined. In addition, in the following embodiments, the same parts are given the same reference numerals, and duplicate explanations will be omitted.

[First embodiment]
FIG. 1 is a schematic diagram of an information processing device according to the first embodiment. As shown in FIG. 1, the information processing device 10 is a so-called wearable device that is worn on the body of a user U. In the example of this embodiment, the information processing device 10 includes a device 10A worn on the eye of the user U, a device 10B worn on the ear of the user U, and a device 10C worn on the arm of the user. The device 10A worn on the eye of the user U includes a display unit 26A (described later) that outputs a visual stimulus to the user U (displays an image), the device 10B worn on the ear of the user U includes a sound output unit 26B (described later) that outputs an auditory stimulus (sound) to the user U, and the device 10C worn on the arm of the user U includes a tactile stimulus output unit 26C (described later) that outputs a tactile stimulus to the user U. However, the configuration of FIG. 1 is an example, and the number of devices and the position of attachment to the user U may be arbitrary. For example, the information processing device 10 is not limited to a wearable device, and may be a device carried by the user U, such as a so-called smartphone or tablet terminal.

FIG. 2 is a block diagram showing an example of the configuration of an information processing device according to the first embodiment. As shown in FIG. 1, the information processing device 10 includes a behavioral state sensor 20, an input unit 22, an output unit 24, a communication unit 26, a memory unit 28, and a control unit 30.

The behavioral state sensor 20 is a sensor that detects behavioral state information related to the behavioral state of the user U wearing the information processing device 10. The behavioral state information of the user U may include various information related to the behavior of the user U. The behavioral state information of the user U may include information related to at least the physical body movements of the user U, the date and time when the behavior is performed, the location where the behavior is performed, and the time when the behavior is performed.

The behavioral state sensor 20 includes a camera 20A, a microphone 20B, a GNSS receiver 20C, an acceleration sensor 20D, a gyro sensor 20E, a light sensor 20F, a temperature sensor 20G, and a humidity sensor 20H. However, the behavioral state sensor 20 may include any sensor that detects behavioral state information, and may include, for example, at least one of the camera 20A, the microphone 20B, the GNSS receiver 20C, the acceleration sensor 20D, the gyro sensor 20E, the light sensor 20F, the temperature sensor 20G, and the humidity sensor 20H, or may include other sensors.

The camera 20A is an imaging device, and captures the surroundings of the information processing device 10 by detecting visible light around the information processing device 10 (user U) as behavioral state information. The camera 20A may be a video camera that captures images at a predetermined frame rate. The position and orientation of the camera 20A in the information processing device 10 are arbitrary, but for example, the camera 20A may be provided in the device 10A shown in FIG. 1, and the imaging direction may be the direction in which the face of the user U is facing. This allows the camera 20A to capture an object in the line of sight of the user U, that is, an object within the range of the user U's field of vision. The number of cameras 20A is arbitrary, and may be one or more. Note that when there are multiple cameras 20A, information on the direction in which the camera 20A is facing is also acquired.

The microphone 20B is a microphone that detects sounds (sound wave information) around the information processing device 10 (user U) as behavioral state information. The position, orientation, and number of microphones 20B installed in the information processing device 10 are arbitrary. Note that, when there are multiple microphones 20B, information on the direction in which the microphones 20B are facing is also acquired.

The GNSS receiver 20C is a device that detects the position information of the information processing device 10 (user U) as behavioral state information. The position information here is earth coordinates. In this embodiment, the GNSS receiver 20C is a so-called GNSS (Global Navigation Satellite System) module, and receives radio waves from satellites to detect the position information of the information processing device 10 (user U).

The acceleration sensor 20D is a sensor that detects the acceleration of the information processing device 10 (user U) as behavioral state information, for example, gravity, vibration, and impact.

The gyro sensor 20E is a sensor that detects the rotation and orientation of the information processing device 10 (user U) as behavioral state information, and detects this using the principles of Coriolis force, Euler force, centrifugal force, etc.

The optical sensor 20F is a sensor that detects the intensity of light around the information processing device 10 (user U) as behavioral state information. The optical sensor 20F can detect the intensity of visible light, infrared light, and ultraviolet light.

The temperature sensor 20G is a sensor that detects the temperature around the information processing device 10 (user U) as behavioral status information.

The humidity sensor 20H is a sensor that detects the humidity around the information processing device 10 (user U) as behavioral status information.

The input unit 22 is a device that accepts user operations and may be, for example, a touch panel.

The output unit 24 outputs the output result by the information processing device 10. The output unit 24 has, for example, a display unit 24A that displays video and an audio output unit 24B that outputs audio. In this embodiment, the display unit 24A is, for example, a so-called HMD (Head Mounted Display). The audio output unit 24B is a speaker that outputs audio.

The communication unit 26 is a module that communicates with an external device, and may include, for example, an antenna. In this embodiment, the communication method used by the communication unit 26 is wireless communication, but the communication method may be any method.

The storage unit 28 is a memory that stores various information such as the calculation contents and programs of the control unit 30, and includes at least one of a main storage device such as a RAM (Random Access Memory) or a ROM (Read Only Memory), and an external storage device such as a HDD (Hard Disk Drive).

The memory unit 28 stores a learning model 28A and map data 28B. The learning model 28A is an AI model used to identify the environment in which the user U is located based on environmental information. The map data 28B is data including location information of real buildings and natural objects, and can be said to be data in which the earth coordinates are associated with real buildings and natural objects. Processing using the learning model 28A and the map data 30B will be described later. The learning model 28A, the map data 28B, and the program for the control unit 30 stored in the memory unit 28 may be stored in a recording medium readable by the information processing device 10. In addition, the program for the control unit 30 stored in the memory unit 28, the learning model 28A, and the map data 28B are not limited to being stored in advance in the memory unit 28, and the information processing device 10 may acquire these data from an external device by communication when using these data.

The control unit 30 controls the operation of each unit of the information processing device 10. The control unit 30 is realized, for example, by a CPU (Central Processing Unit) or MPU (Micro Processing Unit) executing a program stored in a storage unit (not shown) using a RAM or the like as a working area. The control unit 30 may be realized, for example, by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array). The control unit 30 may be realized by a combination of hardware and software.

The control unit 30 includes a behavioral state information acquisition unit 40, a behavioral pattern information generation unit 42, a behavioral score calculation unit 44, a behavioral pattern identification unit 46, a memory control unit 48, and an output control unit 50.

The behavioral state information acquisition unit 40 controls the behavioral state sensor 20 to cause the behavioral state sensor 20 to detect behavioral state information of the user U. The behavioral state information acquisition unit 40 acquires the behavioral state information detected by the behavioral state sensor 20.

The behavior pattern information generating unit 42 generates behavior pattern information based on the behavior state information acquired by the behavior state information acquiring unit 40. The behavior pattern information generating unit 42 generates behavior pattern information in a multidimensional space having, for example, the coordinate axes of parameters of at least the date, time, and place at which the behavior state of the user U was detected, based on the behavior state information.

The behavior score calculation unit 44 calculates a behavior score based on the behavior pattern information generated by the behavior pattern information generation unit 42. For example, the behavior score calculation unit 44 groups the behavior pattern information groups, in which behavior state information is collected, into spaces where the density of the behavior pattern information groups exceeds a predetermined density. The behavior score calculation unit 44 calculates a behavior score for the behavior pattern information groups based on the spaces that include the grouped behavior pattern information groups. Specifically, the behavior score calculation unit 44 calculates, for example, information on the size of the space that includes the behavior pattern information groups as the behavior score.

The behavior pattern identification unit 46 identifies the behavior pattern of user U based on the behavior score calculated by the behavior score calculation unit 44. The behavior pattern identification unit 46 determines that the behavior corresponding to the behavior pattern information group in which the behavior score value exceeds a predetermined threshold is the behavior pattern of user U. The behavior pattern identification unit 46 identifies the type of behavior that user U was performing based on, for example, image data, voice data, position information, acceleration information, posture information, infrared and ultraviolet intensity information, temperature information, and humidity information acquired by the behavior state information acquisition unit 40.

The memory control unit 48 controls the memory unit 28 to perform storage. Information relating to the behavior pattern of user U identified by the behavior pattern identification unit 46 is stored in the memory unit 28. The memory control unit 48 stores information relating to the behavior pattern of user U identified by the behavior pattern identification unit 46 in the memory unit 28 in a predetermined format. The predetermined format will be described later.

The output control unit 50 controls the output unit 24 to perform output. The output control unit 50, for example, controls the display unit 24A to display information related to behavior patterns. The output control unit 50, for example, controls the audio output unit 24B to output information related to behavior patterns by audio.

[Processing content]
The processing contents of the information processing device 10 according to the first embodiment will be described with reference to Fig. 3. Fig. 3 is a flowchart showing an example of the flow of processing of the information processing device 10 according to the first embodiment.

The control unit 30 acquires behavioral state information related to the behavior of the user U from the behavioral state sensor 20 (step S10). Specifically, the behavioral state information acquisition unit 40 acquires image data of the surroundings of the information processing device 10 (user U) from the camera 20A. The behavioral state information acquisition unit 40 acquires audio data of the surroundings of the information processing device 10 (user U) from the microphone 20B. The behavioral state information acquisition unit 40 acquires position information of the information processing device 10 (user U) from the GNSS receiver 20C. The behavioral state information acquisition unit 40 acquires acceleration information of the information processing device 10 (user U) from the acceleration sensor 20D. The behavioral state information acquisition unit 40 acquires posture information of the information processing device 10 (user U) from the gyro sensor 20E. The behavioral state information acquisition unit 40 acquires infrared and ultraviolet intensity information of the surroundings of the information processing device 10 (user U) from the light sensor 20F. The behavioral state information acquisition unit 40 acquires temperature information around the information processing device 10 (user U) from the temperature sensor 20G. The behavioral state information acquisition unit 40 acquires humidity information around the information processing device 10 (user U) from the humidity sensor 20H. The behavioral state information acquisition unit 40 acquires these pieces of information sequentially at predetermined intervals. The behavioral state information acquisition unit 40 may acquire each piece of behavioral state information at the same timing or at different timings. In addition, the predetermined period until the next behavioral state information is acquired may be set arbitrarily, and the predetermined period may be the same or different for each piece of environmental information.

The behavior of user U may include three elements: the physical body movements of the behavior itself, the date and time the behavior is performed, the place where the behavior is performed, and the time the behavior is performed at that place. The behavior of user U may include not only the body movements of user U, but also actions such as "playing golf," "watching a movie," and "shopping." Even if the body movements of user U acquired by the behavioral state information acquisition unit 40 are the same, the actions being performed may be different if the position information is different.

The control unit 30 generates behavior pattern information of the user U (step S11). Specifically, the behavior pattern information generation unit 42 generates behavior pattern information of the user U based on the behavior state information of the user U acquired by the behavior state information acquisition unit 40. The control unit 30 groups the behavior pattern information group (step S12). Specifically, the behavior pattern information generation unit 42 groups the behavior pattern information group by space where the density of the behavior pattern information group where the behavior pattern information is collected exceeds a predetermined density. The control unit 30 calculates a behavior score (step S13). Specifically, the behavior score calculation unit 44 calculates the distance from the center to the edge of the space including the behavior pattern information group as the behavior score. In other words, the behavior score calculation unit 44 calculates the size of the space including the behavior pattern information group as the behavior score.

Figure 4 is a diagram explaining the multidimensional space in which behavior pattern information is generated.

Figure 4 shows a three-dimensional space with date, time, and location as coordinate axes. In Figure 4, date and time is from midnight to midnight, location is the one-dimensional linear distance from home, and time is the time from when an action is determined to start to when it is determined to end, but this is not limited to this. For example, location may be a name and address, etc.

The behavior pattern information generating unit 42 generates behavior pattern information by plotting points P at predetermined time intervals in the three-dimensional space shown in FIG. 4. The behavior pattern information generating unit 42 plots points P at one-minute intervals, but is not limited to this. For example, the behavior pattern information generating unit 42 assumes that the user U watched a movie for "two hours" in "near area A2" at "around noon". In this case, the behavior pattern information generating unit 42 plots points P at predetermined intervals from the point where "around noon", "near area A2", and "two hours" intersect, parallel to the axis of time and date, to "around 14:00". The behavior pattern identifying unit 46 can identify that the user U was in a movie theater based on image data, voice data, position information, acceleration information, posture information, infrared and ultraviolet intensity information, temperature information, and humidity information of the user U's surroundings. In the case of "purchasing groceries" as the behavior of the user U, there are usually multiple candidates for grocery stores such as supermarkets and underground shopping malls in department stores.

The behavior pattern information generating unit 42 groups, in the three-dimensional space shown in FIG. 4, spaces in which the density of the behavior pattern information group in which the points P are generated as behavior pattern information exceeds a predetermined density as the same behavior pattern. For example, the behavior pattern information generating unit 42 performs scanning using the unit space US for the space S in which the behavior pattern information can be generated. For example, the behavior pattern information generating unit 42 counts the number of points P included in the unit space US at each location in the space S. For example, the behavior pattern information generating unit 42 identifies the location in which the number of points P included in the unit space US is the largest as the center of the behavior pattern information. The behavior pattern information generating unit 42 counts the number of points P included in the unit space US surrounding the unit space US with the largest number of points P, and identifies the unit space US containing about 60% of the points P of the unit space US with the largest number of points P as the same group G. The behavior score calculating unit 44 calculates, for example, the length of a perpendicular line drawn from the center of the group G to any surface as the behavior score. The behavior score calculation unit 44 calculates, for example, the volume of the group G as the behavior score. Note that the behavior pattern information generation unit 42 may identify multiple groups G in the three-dimensional space shown in FIG. 4.

Here, the behavior of the user U may include behavior of purchasing items that are not purchased frequently, such as "going to a dealer" for "purchasing a car" and "going to a home improvement center" for "purchasing a chair". It is considered that these behaviors are plotted in the three-dimensional space shown in FIG. 4 in a random manner in terms of location, time, and date and time. However, for example, if the user U does "exercises" for "about 10 minutes" at 6 a.m. every day in a nearby "park", the points plotted in the three-dimensional space become denser in space with each passing day. Even in the same park, the place where the exercises are done may differ from day to day by several meters to several tens of meters, and the time may vary from 10 minutes to about ±2 minutes. For this reason, the behavior pattern information generating unit 42 can generate behavior pattern information by plotting it so that it has a bias like a relatively large dense group in the three-dimensional space. In this way, by selecting parameters such as time, place, and time, which are relatively simple information, as the coordinate axes, it becomes possible to treat a dense group in a three-dimensional space with a specific fluctuation as one behavior pattern.

In the example shown in FIG. 4, the behavior pattern of user U is shown in a three-dimensional space, but the present invention is not limited to this. In this embodiment, the behavior pattern of user U may be generated in any multidimensional space. For example, the location is set to "straight-line distance from home," but the location may be set to "latitude" and "longitude." In this case, the space in which the behavior pattern information is generated is a four-dimensional space. Furthermore, the date and time are shown as "0:00 to 24:00" for one day, but an axis having a scalar value of seven units as "day of the week" may be added to make it a five-dimensional space. In this case, for example, if user U works from Monday to Friday and Saturday and Sunday are holidays, the plot of the behavior pattern information group may change significantly between Monday to Friday and Saturday and Sunday. From the perspective of the "day of the week" axis, it is easier to distinguish between daily behavior patterns and non-daily behavior patterns. Daily behavior patterns and non-daily behavior patterns will be described later.

Note that, although the time is described as the time from the start to the end of the action, the present invention is not limited to this. For example, in the case of an action that is an intermittent, continuing event such as "took 200 practice swings at a bat" or "how many steps walked (or ran)", frequency may be used. For example, if user U has the habit of exercising regularly, by changing the time to frequency, any behavioral pattern can be captured as a parameter of "movement" or "exercise", increasing the possibility of displaying data that is of interest to user U.

Returning to FIG. 3, the control unit 30 determines whether or not the behavior score is less than the threshold value (step S14). Specifically, the behavior pattern identification unit 46 determines whether or not the behavior score calculated by the behavior score calculation unit 44 in step S13 is less than a predetermined threshold value. If it is determined that it is less than the threshold value (step S14; Yes), the process proceeds to step S15. If it is determined that it is not less than the threshold value (step S14; No), the process proceeds to step S18.

If the answer to step S14 is Yes, the control unit 30 identifies the behavior pattern (step S15). Specifically, the behavior pattern identification unit 46 identifies the behavior corresponding to the behavior pattern information group whose behavior score is equal to or less than a predetermined threshold as the behavior pattern of the user U.

The control unit 30 identifies the type of behavioral state of the identified behavior pattern (step S16). Specifically, the behavioral pattern identification unit 46 may identify the type of behavioral state performed by the user U based on the behavioral state information acquired by the behavioral state information acquisition unit 40.

More specifically, the behavior pattern identification unit 46 may identify the behavior state of the user U, for example, using the learning model 28A. The learning model 28A is an AI model constructed by learning a plurality of data sets as teacher data, with the detection result of the behavior state sensor 20 and information indicating the type of behavior state indicated by the detection result of the behavior state sensor 20 being treated as one data set. The behavior pattern identification unit 46 inputs the detection result of the behavior state sensor 20 into the learned learning model 28A, acquires information indicating the type of behavior state indicated by the detection result, and identifies the type of behavior state of the user U. The behavior pattern identification unit 46 uses the learned learning model 28A to identify, for example, that the user U is playing golf, shopping, at a movie theater, etc.

The control unit 30 stores the behavior pattern in the memory unit 28 (step S17). Specifically, the memory control unit 48 records the behavior pattern identified in step S16 in a predetermined format.

Figure 5 is a diagram for explaining the format for storing behavioral patterns. In this embodiment, behavioral patterns can be associated in advance with locations that can predict, for example, 16 to 64 types of behavior. For example, a "golf course" can predict "playing golf," a "movie theater" can predict "watching a movie," a "shopping center" can predict "shopping," and a "park" can predict "doing exercises." In this embodiment, the location information of user U can be obtained from the GNSS receiver 20C of the behavioral state sensor 20, so that the locations visited by user U over a period of several weeks can be registered. This allows this embodiment to sequentially number candidates for behavioral patterns related to user U's lifestyle.

As shown in FIG. 5, storage format F1 may include areas D1, D2, D3, D4, and D5.

In the area D1, a number numbered to the identified behavior pattern is stored. The area D1 is composed of, for example, 3 bytes. The area D2 stores the number of dimensions of the space in which the behavior pattern information is plotted as a group. The area D2 is composed of, for example, 1 byte. In this case, the space can be up to 255 dimensions. In the area D3, the behavior score R of the behavior pattern information group determined to be the behavior pattern of the user U is stored. The behavior score R has a fluctuation of the judgment error of the behavior pattern, and the smaller the value of the behavior score R, the higher the reliability of the behavior pattern. The area D4 is a reserved area. In the area D5, which is the last bit of the last 1-byte area of the reserved area, an identifier indicating whether the behavior pattern is everyday or non-everyday is stored. In the area D5, 0 is written if the behavior pattern is an everyday behavior pattern described later, and 1 is written if the behavior pattern is an non-everyday behavior pattern. The reserved area can be used when you want to add additional information when each behavior pattern occurs. The reserved area may, for example, have an area of 6 bytes or more and may contain numerical information corresponding to N dimensions (N is any integer).

Returning to FIG. 3, the control unit 30 determines whether or not there are other groups (step S18). Specifically, the behavior score calculation unit 44 determines whether or not there are grouped behavior pattern information sets for which behavior scores should be calculated. If it is determined that there are other groups (step S18; Yes), the process proceeds to step S13. If it is determined that there are no other groups (step S18; No), the process proceeds to step S19.

The control unit 30 determines whether or not to end the process (step S19). Specifically, the control unit 30 determines to end the process when an operation to end the process, or an operation to turn off the power, etc. is received. If it is determined that the process is not to be ended (step S19; No), the process proceeds to step S10. If it is determined that the process is to be ended (step S19; Yes), the process of FIG. 3 is ended.

As described above, the information processing device 10 according to the first embodiment detects the behavioral state of the user U and generates a group of behavioral pattern information corresponding to the behavioral state in a multidimensional space. This allows the information processing device 10 according to the first embodiment to identify the behavioral pattern of the user U based on the group of behavioral pattern information.

[Second embodiment]
Next, a second embodiment will be described. Fig. 6 is a flowchart showing an example of a process flow of the information processing device according to the second embodiment. The configuration of the information processing device according to the second embodiment is the same as that of the information processing device 10 shown in Fig. 2, and therefore the description will be omitted.

In the second embodiment, the information processing device 10 determines whether the identified behavior pattern of the user U is a routine daily behavior or a non-routine non-daily behavior.

Figure 7 is a diagram to explain daily and non-daily behaviors.

In FIG. 7, space SA indicates, for example, a range grouped as having the same behavioral pattern. Space SB, for example, has the same center as space SA and its volume is about 60% of space SA.

In the second embodiment, the behavior pattern information included in the space SB is a behavior pattern whose behavior score is less than a first threshold. In the second embodiment, a behavior pattern whose behavior score is less than the first threshold is determined to be a daily behavior pattern. In the example shown in FIG. 7, the behavior pattern corresponding to point P1 in the space SB is determined to be a daily behavior pattern.

In the second embodiment, the behavior pattern included in the space between space SA and space SB is a behavior pattern with a behavior score equal to or greater than a first threshold and less than a second threshold. In the second embodiment, a behavior pattern with a behavior score equal to or greater than a first threshold and less than a second threshold is determined to be an extraordinary behavior pattern. In the example shown in FIG. 7, the behavior pattern corresponding to point P2 in the space between space SA and space SB is determined to be an extraordinary behavior pattern.

In the second embodiment, the behavior patterns included in the space outside the space SA are behavior patterns whose behavior scores are equal to or greater than the second threshold. In the second embodiment, behavior patterns whose behavior scores are equal to or greater than the second threshold are excluded from the user's behavior patterns.

Returning to FIG. 6, the process from step S20 to step S23 in the figure is the same as the process from step S10 to step S13 shown in FIG. 3, so the explanation is omitted.

The control unit 30 determines whether the behavior score is less than the first threshold (step S24). Specifically, the behavior pattern identification unit 46 determines whether the behavior score calculated by the behavior score calculation unit 44 in step S23 is less than a predetermined first threshold. If it is determined that it is less than the first threshold (step S24; Yes), the process proceeds to step S25. If it is determined that it is not less than the first threshold (step S24; No), the process proceeds to step S28.

If the answer to step S24 is Yes, the control unit 30 identifies the behavior pattern of daily behavior (step S25). Specifically, the behavior pattern identification unit 46 identifies the behavior corresponding to the behavior pattern information group whose behavior score is less than a predetermined first threshold as the behavior pattern of daily behavior of the user U.

The control unit 30 identifies the type of behavioral state of the identified daily behavior pattern (step S26). Specifically, the behavior pattern identification unit 46 may identify the type of behavioral state of the daily behavior performed by the user U based on the behavioral state information acquired by the behavioral state information acquisition unit 40.

The control unit 30 stores the behavior pattern of the daily behavior in the memory unit 28 (step S27). Specifically, the memory control unit 48 records the behavior pattern of the daily behavior identified in step S25 in a predetermined format.

If step S24 is determined as No, the control unit 30 determines whether the behavior score is equal to or greater than the first threshold and less than the second threshold (step S28). Specifically, the behavior pattern identification unit 46 determines whether the behavior score calculated by the behavior score calculation unit 44 in step S23 is equal to or greater than a predetermined first threshold and less than a predetermined second threshold. If it is determined to be equal to or greater than the first threshold and less than the second threshold (step S28; Yes), the process proceeds to step S29. If it is determined not to be equal to or greater than the first threshold and less than the second threshold (step S28; No), the process proceeds to step S32.

If step S28 returns Yes, the control unit 30 identifies a behavior pattern of extraordinary behavior (step S29). Specifically, the behavior pattern identification unit 46 determines that the behavior corresponding to the behavior pattern information group whose behavior score is equal to or greater than a predetermined first threshold and less than a predetermined second threshold is an extraordinary behavior pattern.

The control unit 30 identifies the type of behavioral state of the identified behavior pattern of the non-daily behavior (step S30). Specifically, the behavior pattern identification unit 46 may identify the type of behavioral state of the non-daily behavior performed by the user U based on the behavioral state information acquired by the behavioral state information acquisition unit 40.

The control unit 30 stores the behavior pattern of the extraordinary behavior in the memory unit 28 (step S31). Specifically, the memory control unit 48 records the behavior pattern of the extraordinary behavior identified in step S29 in a predetermined format.

The processes in steps S32 and S33 are the same as those in steps S18 and S19 shown in FIG. 3, and therefore will not be described here.

As described above, the information processing device 10 according to the second embodiment determines whether a behavior pattern is a daily behavior or a non-daily behavior based on the behavior score. This allows the information processing device 10 according to the second embodiment to determine whether the same behavior is a daily routine or a new behavior.

Specifically, in the second embodiment, the discrimination between daily and non-daily behaviors can be used to determine whether a behavior pattern that occurs during commuting time on a weekday is a routine behavior or whether the behavior pattern was performed intentionally, and thus whether there is interest. For example, if a person commutes from one station to another at a fixed time every day, walking to the station at the same time is considered to be routine and not an active behavior that is performed out of interest, and the data for that behavior pattern can be ignored in statistical calculations.

[Third embodiment]
Next, an information processing apparatus according to a third embodiment will be described with reference to Fig. 8. Fig. 8 is a block diagram showing an example of the configuration of the information processing apparatus according to the third embodiment.

As shown in FIG. 8, the information processing device 10a differs from the information processing device 10 shown in FIG. 2 in that it includes a biosensor 32 and that the control unit 30a includes a bioinformation acquisition unit 52 and an activity score calculation unit 54.

The behavior of user U is accompanied by physical movements as well as biological information such as the degree of excitement. Therefore, when identifying the behavioral pattern of user U, it is preferable to take into account the psychological state of user U at the time of the behavior. The information processing device 10a calculates an activity score that indicates the degree to which user U is enjoying the behavior.

The biosensor 32 is a sensor that detects biometric information of the user U. The biosensor 32 may be provided at any position as long as it is capable of detecting the biometric information of the user U. The biometric information here is preferably not unchanging, such as a fingerprint, but information whose value changes depending on the state of the user U, for example. Furthermore, the biometric information here is preferably information relating to the autonomic nerves of the user U, that is, information whose value changes regardless of the will of the user U. Specifically, the biosensor 32 includes a pulse wave sensor 32A, and detects the pulse waves of the user U as biometric information. The biosensor 32 may also include an brain wave sensor that detects brain waves of the user U.

Pulse wave sensor 32A is a sensor that detects the pulse wave of user U. Pulse wave sensor 32A may be, for example, a transmission type photoelectric sensor equipped with a light emitting unit and a light receiving unit. In this case, pulse wave sensor 32A may be configured such that the light emitting unit and the light receiving unit face each other across the fingertip of user U, and the light receiving unit receives light that has passed through the fingertip, and measures the pulse waveform by utilizing the fact that the greater the pressure of the pulse wave, the greater the blood flow. However, pulse wave sensor 32A is not limited to this and may be of any type that can detect pulse waves.

The biometric information acquisition unit 52 controls the biometric sensor 32 to cause the biometric sensor 32 to detect biometric information. The biometric information acquisition unit 52 acquires the biometric information detected by the biometric sensor 32.

The activity score calculation unit 54 calculates the autonomic nerve activity level based on the acquired bio-information. A method for calculating the autonomic nerve activity level will be described later. The activity score calculation unit 54 calculates the activity score. The activity score calculation unit 54 calculates the activity score based on the behavior score calculated by the behavior score calculation unit 44, the behavior pattern identified by the behavior pattern identification unit 46, and the autonomic nerve activity level.

[Processing content]
The process contents of the information processing device 10a according to the third embodiment will be described with reference to Fig. 9. Fig. 9 is a flowchart showing an example of the process flow of the information processing device 10a according to the third embodiment.

The process of step S40 is the same as step S10 shown in FIG. 3, so the explanation is omitted.

The control unit 30a acquires biometric information of the user U (step S41). Specifically, the biometric information acquisition unit 52 controls the pulse wave sensor 32A of the biosensor 32 to acquire pulse wave information of the user U. In this embodiment, as described below, the pulse wave information of the user U is used to calculate an autonomic nerve activity level, which is an indicator of the user U's psychological levels of stress, relaxation, interest, and concentration.

The processes from step S42 to step S47 are the same as the processes from step S11 to step S17 shown in FIG. 3, so their explanations are omitted.

The control unit 30a calculates the activity score (step S48). Specifically, the activity score calculation unit 54 calculates the activity score of the user U based on the pulse wave information acquired in step S41.

The pulse wave will be explained using FIG. 10. FIG. 10 is a graph showing an example of a pulse wave. As shown in FIG. 10, the pulse wave has a waveform in which a peak called an R wave WR appears at a predetermined time interval. Pulse pulsation occurs due to spontaneous firing of pacemaker cells in the sinus node of the heart. The pulse rhythm is strongly influenced by both the sympathetic and parasympathetic nerves. The sympathetic nerves promote cardiac activity. The parasympathetic nerves suppress cardiac activity. Normally, the sympathetic nerves and the parasympathetic nerves act in opposition to each other. At rest or in a state close to rest, the parasympathetic nerves dominate. Normally, the pulse rate increases when adrenaline is secreted due to sympathetic nerve stimulation, and decreases when acetylcholine is secreted due to parasympathetic nerve stimulation. Therefore, it is considered useful to examine the fluctuation of the R-R interval in the electrocardiogram in an autonomic nerve function test. This is described in Fujimoto et al.'s "Normal Reference Values and Standard Prediction Formulas for Autonomic Nerve Function Tests Using Fluctuations in Electrocardiogram R-R Intervals" (Autonomic Nerves, Vol. 30, No. 2, 1987, pp. 167-173) and Hayano et al.'s "Heart Rate Fluctuations and Autonomic Nerve Function" (Biophysics, 28-4, pp. 32-36, 1988). The R-R interval is the interval between consecutive R waves WR in a time series, as shown in FIG. 10. Heart rate fluctuations are measured by treating the R wave, which is the peak of the QPS wave in the signal waveform, as one pulse. Fluctuations in the intervals between R waves in an electrocardiogram, that is, the fluctuations in the time intervals of the R-R intervals between R waves in FIG. 10, are used as an autonomic nerve index. The validity of using the fluctuations in the time intervals of the R-R intervals as an autonomic nerve index has been reported by many medical institutions. The fluctuations in the R-R intervals are large when at rest and small when under stress.

There are several characteristic fluctuations in the R-R interval. One is a low-frequency component that appears around 0.1 Hz, which originates from the modulation of sympathetic nervous activity accompanying feedback regulation of vascular blood pressure. The other is a high-frequency component that is a modulation synchronized with breathing and reflects respiratory sinus arrhythmia. The high-frequency component reflects direct interference with the preganglionic neurons of the vagus nerve by the respiratory center and the antireceptor reflex of pulmonary stretch receptors and blood pressure changes due to breathing, and is considered to be a parasympathetic nerve index that mainly affects the heart. In other words, among the waveform components that measure the fluctuation between the R-R waves of the pulse wave, the power spectrum of the low-frequency component indicates the activity of the sympathetic nerve, and the power spectrum of the high-frequency component indicates the activity of the parasympathetic nerve.

The fluctuation of the input pulse wave is found by the differential value of the R-R interval value. In this case, if the differential value of the R-R interval is not data with equal intervals in time, the activity score calculation unit 54 converts it into equal interval time series data using three-dimensional spline interpolation or the like. The activity score calculation unit 54 performs an orthogonal transformation on the differential value of the R-R interval using a fast Fourier transform or the like. As a result, the activity score calculation unit 54 calculates the power spectrum of the high frequency component and the power spectrum of the low frequency component of the differential value of the R-R interval value of the pulse wave. The activity score calculation unit 54 calculates the sum of the power spectrum of the high frequency component as RRHF. The activity score calculation unit 54 calculates the sum of the power spectrum of the low frequency component as RRLF. The activity score calculation unit 54 calculates the autonomic nerve activity using formula (1). The activity score calculation unit 54 is also called an autonomic nerve activity calculation unit.

In equation (1), AN is the autonomic nerve activity, RRHF is the sum of the power spectrum of the high frequency components, and RRLF is the sum of the power spectrum of the low frequency components. C1 and C2 are fixed values defined to suppress divergence of the AN solution.

The activity score calculation unit 54 calculates the activity score using formula (2).

In formula (2), NS is the activity score, AP is the behavior pattern, R is the behavior score, and AN is the autonomic nerve activity level. That is, the activity score calculation unit 54 may calculate the activity score using a function that includes the behavior pattern, the behavior score, and the autonomic nerve activity level as parameters.

The activity score calculation unit 54 may also calculate the activity score using, for example, a learning model. The learning model is an AI model constructed by learning a plurality of data sets as teacher data, with the behavior pattern, the behavior score, the autonomic nerve activity, and the activity score as one data set. In this case, the activity score calculation unit 54 inputs the behavior pattern, the behavior score, and the autonomic nerve activity into the learned learning model, acquires information indicating the activity score, and calculates the activity score.

The control unit 30 presents the activity score to the user U (step S49). Specifically, the output control unit 50 controls at least one of the display unit 24A and the audio output unit 24B to present the activity score to the user U.

The control unit 30 stores the behavior pattern and activity score in the memory unit 28 (step S50). Specifically, the memory control unit 48 records the behavior pattern and activity score identified in step S46 in a predetermined format.

Figure 11 is a diagram for explaining the format for storing behavioral patterns. As shown in Figure 11, storage format F2 may include areas D1a, D2a, D3a, D4a, D5a, and D6a.

Area D1a stores a numbered value for the identified behavior pattern. Area D1a may be composed of, for example, 3 bytes. Area D2a stores the number of dimensions of the space in which the behavior pattern information is plotted as a group. Area D2a may be composed of, for example, 1 byte. Area D3a stores the behavior score R of the behavior pattern information group determined to be the behavior pattern of user U. Area D4a stores the autonomic nerve activity level. Area D4a may be composed of, for example, 2 bytes. Area D5a stores the activity level score. Area D5 may be composed of, for example, 2 bytes. Area D6a is a reserved area.

The processes in steps S51 and S52 are the same as those in steps S18 and S19 shown in FIG. 3, and therefore will not be described here.

As described above, the information processing device 10a according to the third embodiment can calculate an activity score that indicates the degree to which the user enjoyed an action identified as the user U's behavior pattern when the user was performing that action. This allows the information processing device 10a according to the third embodiment to more appropriately identify the behavior pattern of the user U.

[Fourth embodiment]
Next, an information processing apparatus according to a fourth embodiment will be described with reference to Fig. 12. Fig. 12 is a block diagram showing an example of the configuration of the information processing apparatus according to the fourth embodiment.

As shown in FIG. 12, the information processing device 10b differs from the information processing device 10a shown in FIG. 8 in that the memory unit 28a stores correction data 28C and the control unit 30b includes an activity score correction unit 56.

When the country or region changes, the same behavioral pattern may be perceived differently due to differences in sensibility, etc. The information processing device 10b corrects the activity score depending on the country or region.

The correction data 28C is data used by the activity score correction unit 56 when correcting the activity score. The correction data 28C is data that associates, for example, a behavioral pattern with a correction coefficient that is multiplied by the activity score depending on the country or region.

Figure 13 is a diagram for explaining an example of correction data. Figure 13 shows action patterns MP1, MP2, and MP3 as action patterns. Also, areas A1, A2, and A3 are shown as countries or regions. In Figure 13, action patterns are shown conceptually, such as action pattern MP1, but in practice they are shown specifically, such as "playing golf." Also, countries or regions are shown conceptually, such as area A1, but in practice they are shown as specific country names, such as Japan, or specific region names, such as Tokyo.

As shown in FIG. 13, in area A1, the activity score of behavior pattern MP1 is multiplied by 0.5, the activity score of behavior pattern MP2 is multiplied by 0.2, and the activity score of behavior pattern MP3 is multiplied by 0.1. In area A2, the activity score of behavior pattern MP1 is multiplied by 0.2, the activity score of behavior pattern MP2 is multiplied by 0.6, and the activity score of behavior pattern MP3 is multiplied by 0.9. In area A3, the activity score of behavior pattern MP1 is multiplied by 0.3, the activity score of behavior pattern MP2 is multiplied by 0.7, and the activity score of behavior pattern MP3 is multiplied by 0.5. As shown in FIG. 13, if the country or region is different, the correction coefficient multiplied by the activity score may change even if the behavior pattern is the same. The correction coefficient is generated by conducting a questionnaire survey on possible behavior patterns for each country or region. In FIG. 13, the correction coefficients are classified by area, such as area A1 to area A3, but the correction coefficients may be classified by a predetermined condition other than area. For example, depending on the type of behavioral pattern to be targeted, the correction coefficients may be classified by age, gender, or the like.

The activity score correction unit 56 corrects the activity score calculated by the activity score calculation unit 54. Specifically, the activity score correction unit 56 corrects the activity score using correction data 28C based on the country or region in which the behavior pattern is identified.

[Processing content]
The process of the information processing device 10b according to the fourth embodiment will be described with reference to Fig. 14. Fig. 14 is a flowchart showing an example of the process flow of the information processing device 10b according to the fourth embodiment.

The processes from step S60 to step S68 are the same as the processes from step S40 to step S48 shown in FIG. 9, so their explanations are omitted.

The control unit 30b corrects the activity score (step S69). Specifically, the activity score correction unit 56 corrects the activity score calculated by the activity score calculation unit 54 using the correction data 28C based on the location information identified by the behavior pattern identification unit 46.

The control unit 30b presents the corrected activity score to the user U (step S70). Specifically, the output control unit 50 controls at least one of the display unit 24A and the audio output unit 24B to present the corrected activity score to the user U.

The control unit 30b stores the behavior pattern and the corrected activity score in the memory unit 28 (step S71). Specifically, the memory control unit 48 records the behavior pattern identified in step S66 and the corrected activity score in a predetermined format.

The processing in steps S72 and S73 is the same as the processing in steps S51 and S52 shown in FIG. 9, and therefore will not be described.

As described above, the information processing device 10b according to the fourth embodiment corrects the activity score by multiplying the activity score by a correction coefficient depending on the country or region. This allows the information processing device 10b according to the fourth embodiment to more appropriately correct the activity score depending on the country or region.

[Modification of the fourth embodiment]
Next, a modified example of the fourth embodiment will be described. In the fourth embodiment, as shown in FIG. 13, the activity score is corrected using correction data 28C in which a behavior pattern is associated with an area. However, the correction data does not have to be associated with a behavior pattern. In other words, a dedicated correction coefficient may be defined for each area. In this case, the activity score correction unit 56 may correct the autonomic nerve activity calculated by the activity score calculation unit 54 based on the correction data. In this case, the activity score correction unit 56 may also be called an autonomic nerve activity correction unit.

Figure 15 is a diagram illustrating an example of correction data. As shown in Figure 15, in the correction data 28Ca, a correction coefficient is defined for each area. In the example shown in Figure 15, the correction coefficient for area A1 is 0.5, the correction coefficient for area A2 is 0.2, and the correction coefficient for area A3 is 0.3.

For example, when it is determined that the area in which the autonomic nerve activity of user U is calculated is area A1, the activity score correction unit 56 corrects the autonomic nerve activity by multiplying the calculated autonomic nerve activity by 0.5. This makes it possible to more appropriately calculate the autonomic nerve activity for each area in the modified example of the fourth embodiment.

[Fifth embodiment]
Next, a fifth embodiment will be described. Fig. 16 is a flowchart showing an example of a process flow of the information processing device according to the fifth embodiment. The configuration of the information processing device according to the fifth embodiment is the same as that of the information processing device 10b shown in Fig. 2, and therefore the description will be omitted.

In the fifth embodiment, the information processing device 10b calculates an activity score independently for the identified daily behavior pattern and non-daily behavior pattern of the user U. Also, in the fifth embodiment, the information processing device 10b independently corrects the calculated activity score for the daily behavior pattern and the non-daily behavior pattern of the user U.

The processes from step S80 to step S84 are the same as the processes from step S40 to step S44 shown in FIG. 9, so their explanations are omitted.

The processes from step S85 to step S87 are the same as the processes from step S24 to step S26 shown in FIG. 6, so their explanation will be omitted.

The control unit 30b calculates the activity score of the behavior pattern of daily behavior (step S88). Specifically, the activity score calculation unit 54 calculates the activity score of the behavior pattern of daily behavior of the user U based on the pulse wave information acquired in step S81.

The control unit 30b corrects the activity score of the behavior pattern of daily behavior (step S89). Specifically, the activity score correction unit 56 corrects the activity score of the behavior pattern of daily behavior calculated by the activity score calculation unit 54 using the correction data 28C based on the location information identified by the behavior pattern identification unit 46.

The control unit 30b presents the activity score of the daily activity pattern after the correction to the user U (step S90). Specifically, the output control unit 50 controls at least one of the display unit 24A and the audio output unit 24B to present the activity score after the correction to the user U.

The control unit 30b stores the daily behavior pattern and the corrected activity score in the memory unit 28 (step S91). Specifically, the memory control unit 48 records the daily behavior pattern identified in step S66 and the corrected activity score in a predetermined format.

If step S85 returns No, the process proceeds to step S92. Steps S92 to S94 are the same as steps S28 to S30 shown in FIG. 6, and therefore will not be described here.

The control unit 30b calculates the activity score of the behavior pattern of non-daily behavior (step S95). Specifically, the activity score calculation unit 54 calculates the activity score of the behavior pattern of non-daily behavior of the user U based on the pulse wave information acquired in step S81.

The control unit 30b corrects the activity score of the behavior pattern of non-daily behavior (step S96). Specifically, the activity score correction unit 56 corrects the activity score of the behavior pattern of non-daily behavior calculated by the activity score calculation unit 54 using the correction data 28C based on the location information identified by the behavior pattern identification unit 46.

The control unit 30b presents the corrected activity score of the behavior pattern of the non-daily behavior to the user U (step S97). Specifically, the output control unit 50 controls at least one of the display unit 24A and the audio output unit 24B to present the corrected activity score to the user U.

The control unit 30b stores the behavior pattern of the extraordinary behavior and the corrected activity score in the memory unit 28 (step S98). Specifically, the memory control unit 48 records the behavior pattern of the extraordinary behavior identified in step S66 and the corrected activity score in a predetermined format.

The processes in steps S99 and S100 are the same as those in steps S51 and S52 shown in FIG. 9, and therefore will not be described here.

As described above, the information processing device 10b according to the fifth embodiment can independently calculate the activity score when the user is performing a behavior pattern identified as a daily behavior and a non-daily behavior. This allows the information processing device 10a according to the third embodiment to more appropriately identify the behavior pattern of the user U.

In addition, the information processing device 10b according to the fifth embodiment corrects the activity scores of the behavior patterns of daily behavior and the activity scores of the behavior patterns of non-daily behavior by multiplying the activity scores by a correction coefficient depending on the country or region. This allows the information processing device 10b according to the fifth embodiment to more appropriately correct the activity scores depending on the country or region.

Sixth Embodiment
Next, an information processing device according to a sixth embodiment will be described with reference to Fig. 17. Fig. 17 is a block diagram showing an example of the configuration of the information processing device according to the sixth embodiment.

As shown in FIG. 17, the information processing device 10c differs from the information processing device 10a shown in FIG. 8 in that the memory unit 28b stores history data 28D and the control unit 30c includes a history data acquisition unit 58 and a learning unit 60.

Since hobbies and interests may differ from user to user, the activity score value may differ from user to user even when the same behavior is being performed. The information processing device 10c according to the sixth embodiment calculates the activity score using a trained model customized for each user.

History data 28D is data related to the history of activity scores. History data 28D may include information regarding the ranking of activity scores for each user over a specified period of time. Specifically, history data 28D may include information on behavioral patterns that result in activity scores higher than a specified value over a specified period of time. The specified period is, for example, three months, but is not limited to this.

Figures 18A to 18C are diagrams illustrating an example of history data 28D. As shown in Figures 18A to 18C, in history data 28D, rankings, behavioral patterns, behavior scores, and activity scores are associated with each other. Figure 18A is an example of history data for user U1. Figure 18B is an example of history data for user U2. Figure 18C is an example of history data for user U3. Figures 18A to 18C show the behavior patterns with the highest activity scores for users U1 to U3 in a specified period, ranked 1st to 5th, respectively.

As shown in FIG. 18A, user U1's first place is behavior pattern MP4 with a behavior score of 10 and an activity score of 99. User U1's second place is behavior pattern MP3 with a behavior score of 9 and an activity score of 85. User U1's third place is behavior pattern MP1 with a behavior score of 8 and an activity score of 80. User U1's fourth place is behavior pattern MP9 with a behavior score of 7 and an activity score of 58. User U1's fifth place is behavior pattern MP3 with a behavior score of 7 and an activity score of 53.

As shown in FIG. 18B, the first place for user U2 is behavior pattern MP4 with a behavior score of 8 and an activity score of 90. The second place for user U2 is behavior pattern MP3 with a behavior score of 7 and an activity score of 88. The third place for user U2 is behavior pattern MP1 with a behavior score of 8 and an activity score of 79. The fourth place for user U2 is behavior pattern MP8 with a behavior score of 9 and an activity score of 51. The fifth place for user U2 is behavior pattern MP3 with a behavior score of 9 and an activity score of 49.

As shown in FIG. 18C, user U3's first place is behavior pattern MP7 with a behavior score of 10 and an activity score of 89. User U3's second place is behavior pattern MP2 with a behavior score of 6 and an activity score of 71. User U3's third place is behavior pattern MP9 with a behavior score of 7 and an activity score of 68. User U3's fourth place is behavior pattern MP4 with a behavior score of 8 and an activity score of 65. User U3's fifth place is behavior pattern MP3 with a behavior score of 9 and an activity score of 57.

As shown in Figures 18A to 18C, even for the same behavior pattern, the behavior pattern with a high activity score varies depending on the user, and there are individual differences in activity scores.

The history data acquisition unit 58 acquires the history data 28D from the memory unit 28c. Specifically, the history data acquisition unit 58 acquires the history data 28D of the user for whom the activity score is to be calculated.

The learning unit 60 generates a trained model for calculating a user's activity score by learning through machine learning based on the learning data. In this embodiment, the learning unit 60 generates a trained model for calculating the activity score based on, for example, the history data 28D acquired by the history data acquisition unit 58. The learning unit 60 learns, for example, the weights of a DNN (Deep Neural Network) as a trained model for calculating the activity score. The learning unit 60 may learn using, for example, a well-known machine learning method such as deep learning. The learning unit 60 may update the trained model, for example, every time the learning data is updated.

The learning process according to the sixth embodiment will be described with reference to FIG. 19. FIG. 19 is a flowchart showing an example of the flow of the learning process according to the sixth embodiment.

The control unit 30c acquires learning data (step S110). Specifically, the history data acquisition unit 58 acquires history data 28D for a predetermined period of time of the user for whom the activity score is to be calculated from the memory unit 28b. The history data 28D acquired by the history data acquisition unit 58 may include information on at least the ranking, the behavior pattern, the behavior score, and the activity score. The history data acquisition unit 58 acquires, for example, history data 28D from 1st to 1000th place for the past three months.

The control unit 30c executes a learning process (step S111). Specifically, the learning unit 60 uses the history data 28D acquired by the history data acquisition unit 58 to learn and generate a learned model for calculating the user's activity score by machine learning. More specifically, the learning unit 60 generates a learned model by treating the behavior pattern, behavior score, autonomic nerve activity, and activity score as one data set, and learning from multiple (e.g., 1,000) data sets as teacher data. The learning unit 60 generates, for example, a learned model for each user for whom the activity score is to be calculated. That is, in this embodiment, a learned model customized for each user is generated.

The control unit 30c stores the trained model (step S112). Specifically, the learning unit 60 stores the generated trained model in the memory unit 28c.

[Processing content]
The process of the information processing device 10c according to the sixth embodiment will be described with reference to Fig. 20. Fig. 20 is a flowchart showing an example of the process flow of the information processing device 10c according to the sixth embodiment.

The processes from step S120 to step S127 are the same as the processes from step S40 to step S47 shown in FIG. 9, so their explanation is omitted.

The control unit 30c calculates the activity score based on the trained model for the user (step S128). Specifically, the activity score calculation unit 54 calculates the activity score for the user using the trained model customized for the user.

The processes from step S129 to step S132 are the same as the processes from step S49 to step S52 shown in FIG. 9, so their explanation will be omitted.

As described above, the information processing device 10c according to the sixth embodiment calculates the activity scores of users U1 to U3 using a trained model that is customized and generated according to the activity score history for each of users U1 to U3. This allows the information processing device 10c according to the sixth embodiment to calculate the activity scores more appropriately in accordance with the sensibilities of the users.

[Seventh embodiment]
An information processing system according to the seventh embodiment will be described with reference to Fig. 21. Fig. 21 is a diagram for explaining an example of the configuration of the information processing system according to the seventh embodiment.

As shown in FIG. 21, the information processing system 1 according to the seventh embodiment includes a plurality of information processing devices 10c and a server device 100. The information processing devices 10c and the server device 100 are communicatively connected via a network N (e.g., the Internet). That is, the information processing system 1 according to the seventh embodiment has a configuration in which the information processing devices 10c according to the sixth embodiment and the server device 100 are communicatively connected.

In the sixth embodiment described above, a trained model is used to calculate the activity level of a user, which is customized and generated for each user according to the activity score history. In the seventh embodiment, the server device 100 stores the history data of multiple users as shared data, and generates a trained model based on the history data of multiple users whose activity scores and behavioral pattern trends are similar to each other to calculate the activity level of the user.

The configuration of the server device according to the seventh embodiment will be described with reference to FIG. 22. FIG. 22 is a block diagram showing an example of the configuration of the server device according to the seventh embodiment.

As shown in FIG. 22, the server device 100 includes a communication unit 110, a control unit 120, and a storage unit 130. The server device 100 is a so-called cloud server.

The communication unit 110 is realized, for example, by a NIC (Network Interface Card) or a communication circuit. The communication unit 110 is connected to the network N wirelessly or via a wire, and transmits and receives information to and from the information processing device 10c.

The control unit 120 controls the operation of each unit of the server device 100. The control unit 120 is realized, for example, by a CPU, an MPU, or the like executing a program stored in a storage unit (not shown) using a RAM or the like as a working area. The control unit 120 may be realized, for example, by an integrated circuit such as an ASIC or an FPGA. The control unit 120 may be realized by a combination of hardware and software. The control unit 120 includes an acquisition unit 122, a determination unit 124, a request unit 126, and a provision unit 128.

The acquisition unit 122 acquires, for example, history data related to the history of the activity score of each user wearing the information processing device 10c from the communication unit 110. The acquisition unit 122 acquires, for example, history data _28D1 to _28D3 of users U1 to U3 shown in Figures 18A to 18C. The acquisition unit 122 stores the acquired history data in the storage unit 130 as shared data 132.

The determination unit 124 determines the trend of the history data from the shared data 132. For example, the determination unit 124 determines whether there are any users whose history data has similar trends.

When there are multiple users whose history data tendencies are similar, the request unit 126 requests whether or not to allow other users to use the user's history data.

When the use of the history data is permitted, the providing unit 128 provides the history data to users whose history data has similar trends.

The storage unit 130 is a memory that stores various information such as the calculation contents and programs of the control unit 120, and includes at least one of a RAM, a main storage device such as a ROM, and an external storage device such as a HDD.

The storage unit 130 stores shared data 132. The shared data 132 may include history data related to activity scores of a plurality of users wearing the information processing device 10c. The shared data 132 may include, for example, history data _28D1 to _28D3 of users U1 to U3 shown in Figs. 18A to 18C.

[Server Device Processing]
The process of the server device according to the seventh embodiment will be described with reference to Fig. 23. Fig. 23 is a flowchart showing an example of the flow of the process of the server device according to the seventh embodiment.

The control unit 120 refers to the shared data 132 and determines whether there is a user whose history data is similar (step S140). For example, the shared data 132 includes the history data _28D1 to _28D3 of the user U1 to the user U3 shown in FIG. 18A to FIG. 18C. The determination unit 124 determines that the users whose history data is similar are, for example, those whose behavior patterns from 1st to 3rd place are the same, whose behavior scores for each behavior pattern are 8 or more, and whose activity score difference is 10 or less. In this case, the determination unit 124 specifies that the 1st activity score of the user U1 and the user U2 is the behavior pattern MP4, the 2nd activity score is the behavior pattern MP3, and the 3rd activity score is the behavior pattern MP1. The determination unit 124 specifies that the behavior score of the behavior pattern MP4 is 10, the behavior score of the behavior pattern MP3 is 9, and the behavior score of the behavior pattern MP1 is 8 for the user U1. The determination unit 124 determines that, for user U2, the behavior score of behavior pattern MP4 is 8, the behavior score of behavior pattern MP3 is 8, and the behavior score of behavior pattern MP1 is 8. The determination unit 124 determines that, for users U1 and U2, the difference in activity score of behavior pattern MP1 is 9, the difference in activity score of behavior pattern MP3 is 3, and the difference in activity score of behavior pattern MP1 is 1. In this case, the determination unit 124 determines that the history data of users U1 and U2 are similar to each other.

In this embodiment, the determination unit 124 has been described as selecting two out of three users, user U1 to user U3, but this is merely an example, and there is no particular limit to the number of users in the actual user population. The determination unit 124 may determine that the history data of three or more users is similar. The determination unit 124 may determine whether the history data is similar by a method other than the method described in this embodiment. The determination unit 124 may determine whether the history data is similar according to a predetermined mathematically defined conditional expression, for example.

If it is determined that there is a similar user (step S140; Yes), proceed to step S141. If it is determined that there is no similar user (step S140; No), end the process of FIG. 23.

If step S140 returns Yes, the control unit 120 requests permission to share from users whose history data is similar (step S141). Specifically, the request unit 126 transmits a notification to users U1 and U2 via the communication unit 110, requesting permission to share the history data.

The control unit 120 determines whether the sharing request has been permitted (step S142). Specifically, the request unit 126 determines whether or not user U1 or user U2 has responded to the request for permission to share the history data sent in step S141, indicating that the sharing of the history data is permitted. If it is determined that the sharing request has been permitted (step S142; Yes), the process proceeds to step S143. If it is determined that the sharing request has not been permitted (step S142; No), the process of FIG. 23 ends.

If the answer to step S142 is Yes, the control unit 120 shares the history data (step S143). Specifically, when the providing unit 128 is permitted to share the history data by the user U2, the providing unit ₁₂₈ transmits the history data 28D1 of the user U1 to the information processing device 10c worn by the user U1 via the communication unit 110. Then, the process of FIG. 23 ends.

The learning process according to the seventh embodiment will be described with reference to Fig. 24. Fig. 24 is a flowchart showing an example of the flow of the learning process according to the seventh embodiment. In the following, a process in which the information processing device 10c worn by the user U1 acquires the history data _28D2 of the user U2 and performs the learning process will be described.

The control unit 30c acquires history data (step S150). Specifically, the history data acquisition unit 58 acquires history data _28D1 for a predetermined period of a user for which an activity score is to be calculated from the storage unit 28c. The history data _28D1 acquired by the history data acquisition unit 58 may include information on at least a ranking, a behavior pattern, a behavior score, and an activity score. The history data acquisition unit 58 acquires, for example, history data _28D1 from 1st to 1000th place for the past three months. That is, the history data acquisition unit 58 acquires 1000 sets of data sets as teacher data.

The control unit 30c acquires history data that is similar to the history data of the user U1 from the server device 100 (step S151). Specifically, the history data acquisition unit 58 acquires the history data 28D2 of the user U2 from the server device 100, for example, via the communication unit 26. Here, for example, it may be assumed that there are a small number of behavior patterns included in the 1000 sets of data sets of the history data _28D1 . For example, when generating a trained model, 6000 to 8000 sets or more of data sets may be required. In this embodiment, the history data acquisition unit 58 acquires the history data _28D2 that is similar to the history data _28D1 from the server device 100, thereby supplementing the number of data sets and generating a more optimal trained model.

The control unit 30c executes a learning process (step S152). Specifically, the learning unit 60 uses the history data _28D1 and _28D2 acquired by the history data acquisition unit 58 to learn and generate a trained model for calculating the activity score of the user by machine learning.

The control unit 30c stores the trained model (step S153). Specifically, the learning unit 60 stores the generated trained model in the storage unit 28b. Then, the process of FIG. 24 ends.

[Processing content]
The process of the information processing device 10c according to the seventh embodiment will be described with reference to Fig. 25. Fig. 25 is a flowchart showing an example of the process flow of the information processing device 10c according to the seventh embodiment.

The processes from step S160 to step S167 are the same as the processes from step S120 to step S127 shown in FIG. 20, and therefore will not be described.

The control unit 30c calculates the activity score based on the trained model generated using the shared data (step S168). Specifically, the activity score calculation unit 54 calculates the activity score of the user U1 using the trained model generated using the history data _28D1 and the history data _28D2 .

The processes from step S169 to step S172 are the same as the processes from step S129 to step S132 shown in FIG. 20, and therefore will not be described.

As described above, the information processing device 10c according to the seventh embodiment generates a trained model using the history data _28D1 of the user U1 and the history data _28D2 of the user U2 that is similar to the history data _28D1 , and calculates the activity score using the trained model. This allows the information processing device 10c according to the seventh embodiment to more appropriately calculate the activity score.

[Eighth embodiment]
Next, an information processing device according to the eighth embodiment will be described with reference to Fig. 26. Fig. 26 is a block diagram showing an example of the configuration of the information processing device according to the eighth embodiment.

As shown in FIG. 26, the information processing device 10d differs from the information processing device 10 shown in FIG. 2 in that the output unit 24a includes a tactile stimulation output unit 24C.

Because the user's biometric information changes from moment to moment, the user's activity score may also change in response to changes in the biometric information. The information processing device 10c presents the relative progression of the user's activity score over time, thereby presenting the changes in the activity score in an easy-to-understand manner.

The tactile stimulation output unit 26C is a device that outputs a tactile stimulation to the user U. For example, the tactile stimulation output unit 26C outputs a tactile stimulation to the user by physically operating, such as by vibration, but the type of tactile stimulation is not limited to vibration and may be any type.

The output control unit 50 of the control unit 30d of the information processing device 10d according to the eighth embodiment controls the output unit 24a to output information indicating the temporal change in the activity score calculated by the activity score calculation unit 54. In other words, the output control unit 50 controls the output unit 24a to output information indicating the relative change in the activity score.

Using Figures 27A to 27E, we will explain a method for relatively displaying the temporal progression of the activity score. Figures 27A to 27E are figures for explaining a method for relatively displaying the temporal progression of the activity score.

As shown in FIG. 27A, the output control unit 50, for example, controls the display unit 24A to display the change in activity score over time as graph G1. In FIG. 27A, the horizontal axis represents time and the vertical axis represents the activity score. In graph G1, time t0 represents the point in time when calculation of the activity score began. By visually checking graph G1, the user can easily grasp the change in activity score over time.

As shown in FIG. 27B, the output control unit 50, for example, controls the display unit 24A to display the activity score value at the start and the current activity score value side by side. In the example shown in FIG. 27B, the activity score at the start is 56, and the current activity score is 78. This allows the user to easily understand the change in the activity score over time.

27C, the output control unit 50 controls the display unit 24A to display, for example, arcs obtained by dividing a circle centered on the lower left corner of the screen into four equal parts. In the example shown in FIG. 27C, the arc C1 indicates the activity score at the start time, and its radius r1 indicates the magnitude of the activity score. In the example shown in FIG. 27C, the arcs C2 and C3 indicate the current activity score. If the current activity score is greater than that at the start time, the output control unit 50 causes the display unit 24A to display the arc C2 with a radius r2 greater than the radius r1 together with the arc C1 with a radius r1. If the current activity score is smaller than that at the start time, the output control unit 50 causes the display unit 24A to display the arc C3 with a radius r3 smaller than the radius r1 together with the arc C1 with a radius r1. This allows the user to easily grasp the temporal change in the activity score.

27D, the output control unit 50 controls the display unit 24A to display bars, for example. In the example shown in FIG. 27D, the bar B1 is displayed in the center of the display unit 24A, but may be displayed in the lower left corner or the lower right corner. In the example shown in FIG. 27D, the bar B1 indicates the activity score at the start time, and its height h1 indicates the size of the activity score. In the example shown in FIG. 27D, the bar B2 and the bar B3 indicate the current activity score. If the current activity score is higher than the start time, the output control unit 50 causes the display unit 24A to display the bar B2 with a height h2 higher than the height h1 together with the bar B1 with the height h1. If the current activity score is lower than the start time, the output control unit 50 causes the display unit 24A to display the bar B3 with a height h3 lower than the height h1 together with the bar B1 with the height h1. This allows the user to easily grasp the time progression of the activity score.

As shown in FIG. 27E, the output control unit 50 controls, for example, the display unit 24A to display graph G2. Graph G2 may be a bar graph. In FIG. 27E, the horizontal axis indicates time and the vertical axis indicates the activity score. In graph G2, time t0 indicates the point in time when calculation of the activity score started. By visually checking graph G2, the user can easily grasp the progress of the activity score over time.

The output control unit 50 may, for example, control the audio output unit 24B or the tactile stimulation output unit 26C to output information indicating the temporal progression of the activity score.

A method for outputting information showing the time progression of the activity score by the audio output unit 24B or the tactile stimulation output unit 24C will be described with reference to FIG. 28. FIG. 28 is a diagram for explaining a method for outputting information showing the time progression of the activity score by the audio output unit 24B or the tactile stimulation output unit 24C.

In FIG. 28, the horizontal axis indicates time and the vertical axis indicates the activity score. The activity score NS1 at time t0 is the activity score at the time when calculation of the activity score begins. In the example shown in FIG. 28, the activity score NS1 serves as the reference. The output control unit 50, for example, sets the volume of the sound to be output from the sound output unit 24B indicating the reference activity score NS1. The output control unit 50, for example, sets the strength of the stimulus (e.g., vibration) to be output from the tactile stimulation output unit 24C indicating the reference activity score NS1.

Here, assume that at time t1, the activity score becomes an activity score NS2, which is greater than the activity score NS1. In this case, the output control unit 50, for example, controls the audio output unit 24B to output a set of an audio corresponding to the activity score NS1 and an audio corresponding to the activity score NS2. The output control unit 50, for example, controls the tactile stimulation output unit 24C to output a set of a stimulus corresponding to the activity score NS1 and an stimulus corresponding to the activity score NS2. The audio corresponding to the activity score NS2 is greater than the audio corresponding to the activity score NS1. The stimulus corresponding to the activity score NS2 is stronger than the stimulus corresponding to the activity score NS1. It is preferable that the volume of the audio corresponding to the activity score NS2 is changed according to, for example, the ratio of the activity score SN2 to the activity score NS1. It is preferable that the strength of the stimulus corresponding to the activity score NS2 is changed according to, for example, the ratio of the activity score NS2 to the activity score NS1. This allows the user to understand the relative magnitude of activity score NS2 to activity score NS1.

Here, assume that at time t2, the activity score becomes an activity score NS3, which is smaller than the activity score NS1. In this case, the output control unit 50, for example, controls the audio output unit 24B to output a set of an audio corresponding to the activity score NS1 and an audio corresponding to the activity score NS3. The output control unit 50, for example, controls the tactile stimulation output unit 24C to output a set of a stimulus corresponding to the activity score NS1 and an stimulus corresponding to the activity score NS3. The audio corresponding to the activity score NS3 is smaller than the audio corresponding to the activity score NS1. The stimulus corresponding to the activity score NS3 is weaker than the stimulus corresponding to the activity score NS1. It is preferable that the volume of the audio corresponding to the activity score NS3 is changed according to, for example, the ratio of the activity score SN2 to the activity score NS1. It is preferable that the strength of the stimulus corresponding to the activity score NS3 is changed according to, for example, the ratio of the activity score NS2 to the activity score NS1. This allows the user to understand the relative magnitude of activity score NS2 to activity score NS1.

As described above, the information processing device 10d according to the eighth embodiment presents the temporal change in the user's activity score in a relative manner. This makes it easier to understand the temporal change in the activity score with the information processing device 10d according to the eighth embodiment.

Although the embodiments of the present invention have been described above, the present invention is not limited to the contents of these embodiments. The above-mentioned components include those that a person skilled in the art can easily imagine, those that are substantially the same, and those that are within the so-called equivalent range. Furthermore, the above-mentioned components can be combined as appropriate. Furthermore, various omissions, substitutions, or modifications of the components can be made without departing from the spirit of the above-mentioned embodiments.

10, 10a, 10b, 10c, 10d Information processing device 20 Behavioral state sensor 20A Camera 20B Microphone 20C GNSS receiver 20D Acceleration sensor 20E Gyro sensor 20F Light sensor 20G Temperature sensor 20H Humidity sensor 22 Input unit 24 Output unit 24A Display unit 24B Audio output unit 24C Tactile stimulation output unit 26, 110 Communication unit 28, 130 Memory unit 28A Learning model 28B Map data 28C, 28Ca Correction data 28D History data 30, 30a, 30b, 30c, 30d, 120 Control unit 32 Biometric sensor 32A Pulse wave sensor 40 Behavioral state information acquisition unit 42 Behavioral pattern information generation unit 44 Behavioral score calculation unit 46 Action pattern identification unit 48 Memory control unit 50 Output control unit 52 Biometric information acquisition unit 54 Activity score calculation unit 56 Activity score correction unit 58 History data acquisition unit 60 Learning unit 100 Server device 122 Acquisition unit 124 Determination unit 126 Request unit 128 Provision unit

Claims (5)

A behavioral state sensor that detects behavioral state information related to a behavioral state of a user;
A biosensor for detecting biometric information related to the biometric information of the user;
a behavior pattern information generating unit that generates behavior pattern information in a multidimensional space based on the behavior state information and uses as coordinate axes parameters of at least the date and time, the place, and the time when the behavior state is detected, and groups the behavior pattern information into spaces where the density of the behavior pattern information group exceeds a predetermined density;
a behavior score calculation unit that calculates information regarding a size of a space including the group of behavior pattern information as a behavior score;
a behavior pattern identification unit that identifies the group of behavior pattern information that exists in the space and has a behavior score value equal to or greater than a predetermined value as a behavior pattern of the user;
an autonomic nerve activity calculation unit that calculates an autonomic nerve activity of the user based on the biological information;
an activity score calculation unit that calculates an activity score of the user based on the autonomic nerve activity level, the behavior score, and the behavior pattern;
an output control unit that changes the intensity of an output from an output unit in accordance with the intensity of the autonomic nerve activity;
An information processing device comprising: The output control unit changes the intensity of the output from the output unit in response to a temporal change in the intensity of the autonomic nerve activity of the user.
The information processing device according to claim 1 . The output unit has a tactile stimulation output unit,
The output control unit changes the intensity of the vibration of the tactile stimulus output unit in accordance with the intensity of the autonomic nerve activity.
3. The information processing device according to claim 1 or 2. detecting behavioral state information relating to a behavioral state of a user;
detecting biometric information relating to a biometric profile of the user;
generating behavior pattern information in a multidimensional space based on the behavior state information, the coordinate axes being parameters of at least the date and time, the place, and the time when the behavior state was detected, and grouping the behavior pattern information into spaces where the density of the behavior pattern information group that gathers the behavior pattern information exceeds a predetermined density;
calculating information on the size of a space including the group of behavioral pattern information as a behavior score;
identifying a group of the behavior pattern information that exists in the space and has a behavior score value equal to or greater than a predetermined value as a behavior pattern of the user;
calculating an autonomic nerve activity level of the user based on the biological information;
calculating an activity score of the user based on the autonomic nerve activity level, the behavior score, and the behavior pattern;
changing the intensity of the output from an output unit in accordance with the intensity of the autonomic nerve activity;
An information processing method comprising: detecting behavioral state information relating to a behavioral state of a user;
detecting biometric information relating to a biometric profile of the user;
generating behavior pattern information in a multidimensional space based on the behavior state information, the coordinate axes being parameters of at least the date and time, the place, and the time when the behavior state was detected, and grouping the behavior pattern information into spaces where the density of the behavior pattern information group that gathers the behavior pattern information exceeds a predetermined density;
calculating information on the size of a space including the group of behavioral pattern information as a behavior score;
identifying a group of the behavior pattern information that exists in the space and has a behavior score value equal to or greater than a predetermined value as a behavior pattern of the user;
calculating an autonomic nerve activity level of the user based on the biological information;
calculating an activity score of the user based on the autonomic nerve activity level, the behavior score, and the behavior pattern;
changing the intensity of the output from an output unit in accordance with the intensity of the autonomic nerve activity;
A program that causes a computer to execute the following.

Images