JP7662239B1 - Information processing device, method, program, and system - Google Patents

Abstract

The present invention provides information for planning or instructing exercise therapy, or a signal indicating the physical function of a subject.
[Solution] A program of one aspect of the present disclosure causes a computer to function as: means for acquiring a first exercise intensity when a user is performing a first type of exercise based on first data based on sensing results when the user is performing a first type of exercise; means for acquiring a second exercise intensity when the user is performing a second type of exercise or at rest based on second data based on sensing when the user is performing a second type of exercise different from the first type of exercise or at rest; means for determining a personal index of the user by performing a predetermined calculation on multiple exercise intensities acquired for the user, including the first exercise intensity and the second exercise intensity; and means for calculating parameters representing the user's characteristics related to exercise tolerance based on the user's personal index and a reference value.
[Selected figure] Figure 5

Description

This disclosure relates to an information processing device, a method, a program, and a system.

The importance of exercise therapy, which utilizes exercise to treat and prevent disorders and diseases, is increasing. For example, cardiac rehabilitation aims to help cardiac patients recover their physical strength and self-confidence, return to a comfortable home and social life, and prevent cardiac recurrence or re-hospitalization through a comprehensive activity program that includes exercise therapy. The focus of exercise therapy is aerobic exercise, such as walking, jogging, cycling, and aerobics. To perform aerobic exercise more safely and effectively, it is preferable for the subject to exercise at an intensity close to their own anaerobic metabolic threshold (AT (Anaerobic Threshold)).

The anaerobic metabolic threshold is an example of an evaluation index of exercise tolerance, and corresponds to a change point in the cardiopulmonary function state, that is, an exercise intensity near the boundary between aerobic exercise and anaerobic exercise. The anaerobic metabolic threshold is generally determined by a CPX test (cardiopulmonary exercise stress test), in which the subject is given a gradually increasing exercise load while exhaled gas is collected and analyzed (see Non-Patent Document 1). In a CPX test, the anaerobic metabolic threshold is determined based on the results measured by exhaled gas analysis (e.g., oxygen intake, carbon dioxide output, tidal volume, respiratory rate, minute ventilation, or a combination thereof). In addition to the anaerobic metabolic threshold, the CPX test can also determine the maximum oxygen intake, which corresponds to an exercise intensity near the maximum exercise tolerance.

To ensure the effectiveness and safety of exercise therapy, it is important to properly manage the exercise intensity of the subject.

Patent document 1 describes a method for determining whether or not a subject's ventilatory threshold (VT) has been reached based on the subject's pulsation information, and adjusting the exercise load of the exercise providing device according to the determination result.

JP 2022-059494 A

Saito, Muneyasu, Cardiac Rehabilitation, Physical Therapy, 1997, Vol. 24, No. 8, p. 414-418

In the technical concept described in Patent Document 1, feedback control is performed so that the exercise intensity approaches the ventilatory threshold (VT). However, the exercise intensity for the person exercising depends not only on the type of exercise, but also on the person's physical function and daily physical condition. Therefore, even if this technical concept is applied to exercise therapy, it is not possible to obtain information on the level of exercise intensity that a specified type of exercise will have for the subject, or on what type of exercise should be recommended to the subject to achieve the specified exercise intensity. Furthermore, the technical concept described in Patent Document 1 cannot be simply applied to cases where, for example, a subject is made to exercise at a specified exercise intensity lower than the ventilatory threshold (VT).

The purpose of this disclosure is to provide information to plan or guide exercise therapy, or to provide a signal indicating the subject's physical function.

A program according to one aspect of the present disclosure causes a computer to function as: means for acquiring first data based on a sensing result when a user is performing a first type of exercise; means for acquiring a first exercise intensity when the user is performing the first type of exercise based on the first data; means for acquiring second data based on sensing when the user is performing a second type of exercise different from the first type of exercise or at rest; means for acquiring a second exercise intensity when the user is performing the second type of exercise or at rest based on the second data; means for determining a first individual index of the user by performing a predetermined calculation on a plurality of exercise intensities acquired for the user, including the first exercise intensity and the second exercise intensity; and means for calculating a parameter representing a characteristic of the user related to exercise tolerance based on the user's first individual index and a first reference value.

1 is a block diagram showing a configuration of an information processing system according to an embodiment of the present invention; FIG. 2 is a block diagram showing a configuration of a client device according to the present embodiment. FIG. 2 is a block diagram showing a configuration of a server according to the present embodiment. 1 is a block diagram showing a configuration of a wearable device according to an embodiment of the present invention. FIG. 1 is an explanatory diagram of one aspect of the present embodiment. FIG. 2 is a diagram showing a data structure of an exercise event database according to the present embodiment. FIG. 4 is a diagram showing a data structure of a user profile database according to the present embodiment. FIG. 4 is a diagram illustrating a data structure of a parameter log database according to the present embodiment. 13 is a flowchart of an exercise event recommendation process according to the present embodiment. 11A and 11B are diagrams illustrating an example of a screen displayed in the exercise type recommendation process of the present embodiment. 4 is a flowchart of a parameter monitoring process according to the present embodiment. FIG. 2 is a diagram showing the data structure of a teacher dataset that can be used in this embodiment.

One embodiment of the present invention will be described in detail below with reference to the drawings. In the drawings used to explain the embodiment, the same components are generally designated by the same reference numerals, and repeated explanations will be omitted.

(1) Configuration of the Information Processing System The configuration of the information processing system will be described below. Fig. 1 is a block diagram showing the configuration of the information processing system according to this embodiment.

As shown in FIG. 1, the information processing system 1 includes a client device 10, a server 30, and a wearable device 50.

Here, the number of client devices 10 and wearable devices 50 varies depending on, for example, the number of users. Therefore, the number of client devices 10 and wearable devices 50 may each be two or more. Furthermore, the information processing system 1 may also include a terminal of a person who plans or instructs the exercise therapy. The person who plans or instructs the exercise therapy may include, for example, a medical professional (e.g., a doctor, a nurse, a pharmacist, a physical therapist, an occupational therapist, or a clinical laboratory technician), a nutritionist, or a trainer.

The client device 10 and the server 30 are connected via a network (for example, the Internet or an intranet) NW.
The client device 10 and the wearable device 50 are connected via a wireless channel, for example using Bluetooth (registered trademark) technology.

The client device 10 is an example of an information processing device that sends a request to the server 30. The client device 10 is, for example, a smartphone, a tablet terminal, or a personal computer.

The server 30 is an example of an information processing device that provides the client device 10 with a response in response to a request sent from the client device 10. The server 30 is, for example, a server computer.

The wearable device 50 is an example of an information processing device that can be worn on the user's body (e.g., the arm, hand, or head).

(1-1) Configuration of the Client Device The configuration of the client device will be described below with reference to Fig. 2, which is a block diagram showing the configuration of the client device according to this embodiment.

As shown in FIG. 2, the client device 10 includes a storage device 11, a processor 12, an input/output interface 13, and a communication interface 14. The client device 10 is connected to a display 15, a camera 16, a depth sensor 17, a microphone 18, and an acceleration sensor 19.

The storage device 11 is configured to store programs and data. The storage device 11 is, for example, a combination of Read Only Memory (ROM), Random Access Memory (RAM), and storage (for example, flash memory or a hard disk).

The programs include, for example, the following programs:
-OS (Operating System) programs -Programs for applications that execute information processing (e.g., web browsers, therapeutic apps, rehabilitation apps, or fitness apps) Here, diseases that are the target of therapeutic apps or rehabilitation apps are diseases in which exercise may contribute to improving symptoms, such as heart disease, lifestyle-related diseases (hypertension, diabetes, dyslipidemia, hyperlipidemia), and obesity.

The data includes, for example, the following data:
- Databases referenced in information processing - Data obtained by executing information processing (i.e., the results of executing information processing)

The processor 12 is a computer that realizes the functions of the client device 10 by running a program stored in the storage device 11. The processor 12 is, for example, at least one of the following:
・CPU (Central Processing Unit)
・GPU (Graphic Processing Unit)
・ASIC (Application Specific Integrated Circuit)
・FPGA (Field Programmable Gate Array)

The input/output interface 13 is configured to receive information (e.g., user instructions, images, sounds) from an input device connected to the client device 10, and to output information (e.g., images, commands) to an output device connected to the client device 10.

The input device is, for example, a camera 16, a depth sensor 17, a microphone 18, an acceleration sensor 19, a keyboard, a pointing device, a touch panel, a sensor, or a combination thereof.
The output device is, for example, a display 15, a speaker, or a combination thereof.

The communication interface 14 is configured to control communications between the client device 10 and external devices (eg, another client device 10, a server 30, and a wearable device 50).
Specifically, the communication interface 14 may include a module for communication with the server 30 (e.g., a WiFi module, a mobile communication module, or a combination thereof). The communication interface 14 may include a module for communication with the wearable device 50 (e.g., a Bluetooth module).

The display 15 is configured to display an image (still image or video). The display 15 is, for example, a liquid crystal display or an organic electroluminescence display.

The camera 16 is configured to capture images and generate image signals.

The depth sensor 17 is, for example, a LIDAR (Light Detection And Ranging) sensor. The depth sensor 17 is configured to measure the distance (depth) from the depth sensor 17 to a surrounding object (e.g., a user).

The microphone 18 is configured to receive sound waves and generate a sound signal. The microphone 18 is preferably placed near the user's body (particularly the respiratory tract), for example as an earphone microphone.

The acceleration sensor 19 is configured to detect acceleration.

(1-2) Server Configuration The following describes the server configuration: Fig. 3 is a block diagram showing the server configuration of this embodiment.

As shown in FIG. 3, the server 30 includes a storage device 31, a processor 32, an input/output interface 33, and a communication interface 34.

The storage device 31 is configured to store programs and data. The storage device 31 is, for example, a combination of ROM, RAM, and storage.

The programs include, for example, the following programs:
・OS programs ・Application programs that execute information processing

The data includes, for example, the following data:
- Databases referenced in information processing - Results of information processing

The processor 32 is a computer that realizes the functions of the server 30 by running a program stored in the storage device 31. The processor 32 is, for example, at least one of the following:
CPU
・GPU
・ASIC
・FPGA

The input/output interface 33 is configured to obtain information (eg, a user's instruction) from an input device connected to the server 30 , and to output information to an output device connected to the server 30 .
The input device is, for example, a keyboard, a pointing device, a touch panel, or a combination thereof.
The output device is, for example, a display.

The communication interface 34 is configured to control communication between the server 30 and an external device (e.g., the client device 10).

(1-3) Configuration of the Wearable Device The configuration of the wearable device will be described below. Fig. 4 is a block diagram showing the configuration of the wearable device of this embodiment.

As shown in FIG. 4, the wearable device 50 includes a storage device 51, a processor 52, an input/output interface 53, and a communication interface 54. The wearable device 50 is connected to a display 55, a heart rate sensor 56, and an acceleration sensor 57.

The storage device 51 is configured to store programs and data. The storage device 51 is, for example, a combination of ROM, RAM, and storage.

The programs include, for example, the following programs:
- OS programs - Programs for applications that execute information processing (e.g., medical applications, rehabilitation applications, or fitness applications)

The data includes, for example, the following data:
- Databases referenced in information processing - Results of information processing

The processor 52 is a computer that realizes the functions of the wearable device 50 by running a program stored in the storage device 51. The processor 52 is, for example, at least one of the following.
CPU
・GPU
・ASIC
・FPGA

The input/output interface 53 is configured to acquire information (e.g., user instructions, sensing results) from an input device connected to the wearable device 50, and to output information (e.g., images, commands) to an output device connected to the wearable device 50.

The input device is, for example, a heart rate sensor 56, an acceleration sensor 57, a microphone, a keyboard, a pointing device, a touch panel, or a combination thereof.
The output device is, for example, a display 55, a speaker, or a combination thereof.

The communication interface 54 is configured to control communications between the wearable device 50 and an external device (e.g., the client device 10).
In particular, the communication interface 54 may include a module (eg, a Bluetooth module) for communicating with the client device 10 .

The display 55 is configured to display an image (still image or video). The display 55 is, for example, a liquid crystal display or an organic EL display.

The heart rate sensor 56 is configured to measure the heart rate and generate a sensing signal. As an example, the heart rate sensor 56 measures the heart rate using an optical measurement technique.

The acceleration sensor 57 is configured to detect acceleration.

(2) One aspect of the embodiment One aspect of the present embodiment will be described below. Fig. 5 is an explanatory diagram of one aspect of the present embodiment.

As shown in FIG. 5, the client device 10 and the wearable device 50 sense the user US1 during exercise. The user US sequentially performs a number of exercise types. The combination of a number of exercise types (hereinafter referred to as the "target exercise set") used to calculate the individual difference parameters (described later) may be fixed or may be arbitrarily selected. In the latter case, the exercise types constituting the target exercise set may be determined by the user, the person planning or instructing the exercise therapy for the user, or an algorithm. In addition, the multiple exercise types constituting the target exercise set may be selected from types having different standard exercise intensities. In any case, the information on the standard exercise intensity corresponding to each of the exercise types constituting the target exercise set is stored in an exercise type database described later. The user US1 is typically a person who undergoes exercise therapy, for example, a participant in a (cardiac) rehabilitation program or an exercise instruction program. In the example of FIG. 5, the exercise type is gymnastics, but the exercise type may include any exercise (aerobic exercise or anaerobic exercise).

As an example, the camera 16 captures the appearance (e.g., the whole body) of the user US1 while exercising from the front or diagonal front at a distance of, for example, about 2 m. The camera 16 may be installed at an appropriate height using a tripod or other height adjustment means. The depth sensor 17 measures the distance (depth) from the depth sensor 17 to each part of the user US1. It is also possible to generate three-dimensional video data by combining, for example, video data (two-dimensional) generated by the camera 16 and, for example, depth data generated by the depth sensor 17. The microphone 18 receives sounds (e.g., sounds generated by breathing or speaking) emitted by the user US1 while exercising and generates a sound signal.

The heart rate sensor 56 of the wearable device 50 measures the heart rate of the user US1 while exercising and transmits the measurement results to the client device 10. The acceleration sensor 57 measures the acceleration of the user US1 while exercising and transmits the measurement results to the client device 10.

The client device 10 acquires various sensing data and performs analysis as necessary. As an example, the client device 10 may refer to video data acquired from the camera 16 to analyze the physical condition of the user US1 during exercise. The client device 10 may further refer to depth data acquired from the depth sensor 17 to analyze the physical condition of the user US1 during exercise. The client device 10 transmits user data including at least one of the sensing data or the analysis results of the sensing data to the server 30.

Based on the user data acquired from the client device 10, the server 30 estimates the exercise intensity of the user when performing each of the multiple exercise types constituting the target exercise set. The server 30 then performs a predetermined calculation on the multiple exercise intensities corresponding to each of the multiple exercise types to determine the individual index (described below) of the user US1 for the target exercise set. Furthermore, the server 30 calculates an individual difference parameter that represents the characteristics of the user US1 regarding exercise tolerance (e.g., whether the exercise intensity tends to be high or low compared to a standard person) based on the determined individual index and the reference value (described below) of the individual index for the target exercise set. For example, since the amount of oxygen consumed during exercise depends on muscle mass, there will be individual differences in exercise intensity even for the same exercise type, depending on the muscle development of the user US1 and the part of the body where the load is applied.

The server 30 stores the calculated individual difference parameters in the storage device 31. Using the individual difference parameters, the server 30 can predict the exercise intensity of the user US1 when performing an exercise not included in the target exercise set, or estimate what exercise the user US1 should perform to achieve a desired exercise intensity. Furthermore, by continuously monitoring the individual difference parameters, it is possible to grasp the trend of change (e.g., improvement, maintenance, or decline) in the physical function of the user US1, and the rate of change (e.g., whether the change is rapid or not), and if the physical function of the user US1 is rapidly declining, an alert can be output indicating that an abnormality such as an exacerbation of a heart disease such as heart failure, a physical breakdown, or a temporary deterioration in physical condition is suspected.

In this way, the information processing system 1 calculates individual difference parameters that represent the characteristics of the user US1 regarding exercise tolerance, based on data based on the sensing results of the user US1 while performing each of the exercise events that make up the target event set. Therefore, according to this information processing system 1, such individual difference parameters can be used as a basis for planning or instructing an exercise therapy to be provided to the user US1, or as a signal indicating the physical function or physical condition of the user US1.

(3) Database The database of this embodiment will be described. The following database is stored in the storage device 31.

(3-1) Exercise Event Database The exercise event database of this embodiment will be described below. Fig. 6 is a diagram showing the data structure of the exercise event database of this embodiment.

The exercise type database stores exercise type information. The exercise type information is information about exercise types that are candidates for the recommended exercise type or the target exercise type set. Here, in this embodiment, the exercise types include exercise types that can be performed without using a device that can adjust the exercise load, such as gymnastics, bodyweight training, dancing, walking, running, and treadmills. These exercise types include types performed in a standing position, so there is a wide variety. Furthermore, in this embodiment, the exercise intensity of these exercise types can be adjusted through the form (for example, the positional relationship or angle of parts (which may include parts that are not moved during exercise), the range of motion of parts that are moved during exercise, etc.), pace, number of reps, or the time or number of rest periods. However, the exercise types in this embodiment can also include exercise types performed using a device that can adjust the exercise load, such as an ergometer or strength training using training equipment.

As shown in FIG. 6, the exercise event database includes an "ID" field, a "Name" field, an "Exercise Intensity" field, and an "Exercise Definition" field. Each field is associated with the others.

The "ID" field stores the exercise type ID. The exercise type ID is information that identifies the exercise type that corresponds to the corresponding record.

The "Name" field stores the name of the exercise type. The name of the exercise type is information about the name of the exercise type that corresponds to the record.

The "exercise intensity" field stores exercise intensity information. The exercise intensity information is information related to the standard exercise intensity of the exercise type corresponding to the record. Standard exercise intensity may mean, for example, the exercise intensity when a person with standard physical function performs the corresponding exercise type as defined. As an example, such exercise intensity may be derived by actually measuring the exercise intensity (e.g., average oxygen consumption) when one or more people perform the corresponding exercise type as defined, for example, by performing an exhaled gas analysis (or estimating by analyzing a video of the user exercising), and then statistically processing the results (e.g., averaging).

Note that exercise intensity may be derived by, for example, measuring (or estimating) the exercise intensity for each section that constitutes the corresponding exercise type when one or more people perform the corresponding exercise type as defined, applying the measurement results for each section for each person to a predetermined calculation formula, and then statistically processing (e.g. averaging) the values obtained by this calculation formula between people. Here, a section is a component unit of an exercise type, and when an exercise type is made up of multiple movement patterns, each movement pattern may correspond to a section. For example, when the exercise type is dancing, each choreography may correspond to a section. Also, when the exercise type is squats, the transition from a standing state to a crouching state and the transition from a crouching state to a standing state may each correspond to a section.

The "exercise definition" field stores exercise definition information. The exercise definition information is information regarding the definition of the exercise event corresponding to the corresponding record. The exercise definition information may include information regarding at least one of the following elements.
- Exercise pace (for example, the time it takes to complete one cycle of movement)
Positional relationship of a part (which may be one or more parts) (e.g., foot width, etc.)
Angle of a part (which may be one or more parts) (for example, the direction of the knee, the angle between the upper arm and the forearm)
- Range of motion of the body part (which may be one or more) moved during exercise (for example, the range of motion of each body part during one cycle of movement)
Number of reps, rest time, exercise load (for example, the magnitude of the external load set in the case of exercise performed using a device that can adjust the exercise load, such as an ergometer)

Even for an exercise that is generally recognized as a single event (e.g., squats), multiple exercise events with slightly different exercise intensities can be defined by specifying and defining the pace, form (particularly range of motion), number of repetitions, rest time, and exercise load in detail. For example, multiple exercise events that differ by 0.2 METs can be defined for an exercise that is generally recognized as a single event.

(3-2) User Profile Database The user profile database of this embodiment will now be described with reference to Fig. 7, which is a diagram showing the data structure of the user profile database of this embodiment.

The user profile database stores user profile information. The user profile information is information about the profile of a user of the information processing system 1 (i.e., a person who exercises).

As shown in FIG. 7, the user profile database includes an "ID" field, a "Name" field, a "Target Strength" field, and a "Body" field. Each field is associated with the others.

The "ID" field stores the user ID. The user ID is information that identifies the user corresponding to the corresponding record.

The "Name" field stores user name information. User name information is information about the name of the user corresponding to the record in question (e.g., name, account name, etc.).

The "target intensity" field stores target intensity information (an example of a "predetermined exercise intensity"). The target intensity information is information about the target value of the exercise intensity (for example, oxygen consumption, energy consumption, heart rate, or a combination thereof) set for the user corresponding to the record. As an example, the target intensity information is specified by a person who plans or instructs exercise therapy (for example, a doctor) based on the results of measuring the exercise tolerance of the user, for example, by CPX when using an ergometer (for example, oxygen consumption and heart rate at the anaerobic metabolic threshold (AT)). However, CPX is not essential, and the target value may be specified at the doctor's discretion. As another example, the target intensity information is determined by an algorithm based on the results of measuring the exercise tolerance of the user, for example, by CPX. In order to perform aerobic exercise more safely and effectively, it is preferable to perform exercise at an intensity near the anaerobic metabolic threshold, so the target value of the exercise intensity is, for example, an exercise intensity corresponding to the anaerobic metabolic threshold, but is not limited to this. In addition, an ergometer is usually used as the type of exercise when measuring CPX, but when a treadmill is used, the oxygen consumption at the anaerobic metabolic threshold is about 1.2 to 1.3 times that when an ergometer is used. This is thought to be because the total muscle mass used on the treadmill exceeds the total muscle mass used on the ergometer. Therefore, as a first example, the target value based on the result measured by CPX when using the ergometer may be corrected to about 1.2 to 1.3 times, or a value obtained by further subtracting a predetermined value (for example, 1 METs) may be used as the target intensity information. As a second example, the target value based on the result measured by CPX when using the ergometer may be used as the target intensity information without correction. This makes it possible to prevent the target value from becoming excessively high when an exercise type that uses a relatively small total muscle mass, such as an ergometer, is selected. As a third example, the target value for each exercise type may be corrected, for example, by a coefficient according to the total muscle mass used.

In addition, the doctor may set an upper limit of exercise intensity for the user (an example of exercise prescription). In this case, the user is not allowed to select an exercise type that exceeds the upper limit of the prescribed exercise intensity. For the exercise prescription, a UI (User Interface) screen for exercise prescription may be displayed on the display of the terminal used by the doctor. Such a UI screen may include, for example, the following information:
-User's CPX data -Display area of sample videos of multiple selectable exercise types Here, the display area of the sample videos of each exercise type may be arranged according to the exercise intensity information corresponding to the exercise type. For example, when 3.6 METs is recommended as the upper limit based on the CPX data, a group of display areas of sample videos of exercise types corresponding to 3.4 METs, a group of display areas of sample videos of exercise types corresponding to 3.6 METs, and a group of display areas of sample videos of exercise types corresponding to 3.8 METs may be arranged on the UI screen. Note that the information of the exercise type (e.g., sample videos) may be arranged in units smaller than 0.2 METs or larger.
When the doctor selects one of the display areas, the exercise intensity associated with the corresponding exercise type is set as the upper limit.

In addition, the upper limit specified by the doctor through the exercise prescription may be raised or lowered by a medical professional under the supervision of a doctor during regular (e.g., every two weeks) medical guidance or doctor rounds.

The "Physical" field stores physical information. Physical information is information about the body (functions) of the user corresponding to the record in question. As an example, physical information may include information about the user's age, sex, weight, height, illnesses, etc.

Additionally, the user profile database may store the following information:
Information indicating the person who planned or instructed the user's exercise therapy Information indicating the user's doctor

(3-3) Parameter Log Database The parameter log database of this embodiment will be described below with reference to Fig. 8, which is a diagram showing the data structure of the parameter log database of this embodiment.

The parameter log database may be constructed, for example, for each user of the information processing system 1 (i.e., a person who exercises). Alternatively, the parameter log database may be configured to store records that include information that can identify a user (e.g., a user ID).

The parameter log database stores parameter log information. The parameter log information is information about the log of individual difference parameters calculated for the user.

As shown in FIG. 8, the parameter log database includes a "date" field and a "parameter" field. Each field is associated with the other.

The "Date" field stores date information. The date information is information about the date (or date and time) when the individual difference parameters of the corresponding record were calculated.

The "Parameter" field stores parameter information. Parameter information is information about the values of the individual difference parameters for the corresponding record.

(4) Information Processing The information processing of this embodiment will be described.

(4-1) Exercise type recommendation process The exercise type recommendation process of this embodiment will be described. Fig. 9 is a flowchart of the exercise type recommendation process of this embodiment. Fig. 10 is a diagram showing an example of a screen displayed in the exercise type recommendation process of this embodiment.

The exercise type recommendation process starts when, for example, any one of the following start conditions is met.
- The exercise type recommendation process was called by another process.
The user or a person who plans or instructs the user's exercise therapy performs an operation to call up the exercise type recommendation process.
The client device 10 enters a predetermined state (for example, a predetermined application is started).
The specified date and time has arrived.
A certain amount of time has passed since a certain event.

As shown in FIG. 9, the client device 10 executes acquisition of sensing data (S110).
Specifically, the client device 10 may start taking a video of the user exercising (hereinafter referred to as "user video") by enabling the operation of the camera 16. The client device 10 may also start measuring the distance from the depth sensor 17 to each part of the user exercising (hereinafter referred to as "user depth") by enabling the operation of the depth sensor 17. The client device 10 may start collecting sound (e.g., sound generated by the user's breathing or speaking (hereinafter referred to as "user sound")) by enabling the operation of the microphone 18.

Furthermore, the client device 10 may cause the wearable device 50 to start measuring the heart rate (hereinafter referred to as the "user's heart rate") using the heart rate sensor 56. Furthermore, the client device 10 may enable any sensor (e.g., the acceleration sensor 19 or the acceleration sensor 57) of the client device 10 or the wearable device 50.

The client device 10 then acquires sensing data from each sensor while the user is performing an exercise event (an exercise event constituting a target event set).
Specifically, the client device 10 acquires sensing results generated by the various sensors enabled in step S110. For example, the client device 10 may acquire user video data from the camera 16, user depth data from the depth sensor 17, user sound data from the microphone 18, user heart rate data from the wearable device 50, and user acceleration data relating to the acceleration of the user (hereinafter, "user acceleration") from the acceleration sensor 19 or at least one of the wearable device 50. The client device 10 may also acquire user oxygen intake data relating to the oxygen intake of the user from a device that performs a test on exhaled gas (e.g., a CPX test).

The client device 10 can repeatedly acquire sensing data (S110) while performing one exercise event.

After step S110, the client device 10 executes generation of user data (S111).
Specifically, the client device 10 generates user data based on the sensing data acquired in step S110. The user data may include at least one of the following:
Data acquired in step S111 (e.g., user video data, user depth data, user sound data, user heart rate data, user acceleration data, or user oxygen intake data)
Data obtained by processing the data acquired in step S111 Data obtained by analyzing the user video data (or the user video data and the user depth data) acquired in step S111 (for example, skeletal data, facial expression data, skin color data, breathing data, or a combination thereof, which will be described later)
Information that can identify the type of exercise that the user was performing in step S110 (the type of exercise that constitutes the target exercise set) Information that indicates the user's subjective evaluation of the exercise intensity of the completed exercise type (hereinafter referred to as "perceived exercise intensity").

The client device 10 may generate user data (S111) repeatedly while performing one exercise event, or may generate user data each time an exercise event is completed.

After step S111, the client device 10 transmits the user data (S112).
Specifically, the client device 10 transmits the user data generated in step S111 to the server 30.

The client device 10 may transmit the user data (S112) repeatedly during the performance of one exercise event, may transmit the data each time an exercise event is completed, or may transmit the data only once after all the exercise events constituting the target exercise set have been completed.

After step S112, the server 30 executes estimation of exercise intensity (S130).
Specifically, the server 30 receives the user data transmitted by the client device 10 in step S112. The server 30 estimates the exercise intensity of the user when performing each exercise event constituting the target event set based on the user data acquired from the client device 10. The exercise intensity may be calculated as, for example, energy consumption (e.g., METs), oxygen consumption, exercise intensity based on heart rate (e.g., exercise intensity calculated by the Karvonen method), or a combination thereof. The server 30 may refer to user profile information stored in a user profile database (FIG. 7) to estimate the exercise intensity.

As a first example of estimating exercise intensity (S130), the server 30 estimates the exercise intensity when the user performs one of the exercise types constituting the target exercise set by performing a Karvonen calculation based on the user's heart rate measured over multiple points in time while the user is performing that exercise type and the user's age.

As a second example of estimating exercise intensity (S130), the server 30 uses an estimation model described below to estimate the exercise intensity when the user performs each exercise event that constitutes the target event set. Specifically, the server 30 estimates the exercise intensity by applying the estimation model to input data based on user data (e.g., skeletal data, facial expression data, skin color data, breathing data, heart rate data, or a combination thereof).

As a third example of estimating exercise intensity (S130), the server 30 may calculate the exercise intensity when the user performs each exercise type that constitutes the target exercise set, based on the user's oxygen intake data when performing that exercise type.

As a fourth example of estimating exercise intensity (S130), the server 30 may use the user's audio data to have an expert, such as a physical therapist, conduct a talk test while the user is performing each of the exercise types constituting the target exercise set, and calculate the exercise intensity when the user performs each of the exercise types based on the results of the talk test.

As a fifth example of estimating exercise intensity (S130), the server 30 may obtain information on the perceived intensity of exercise (e.g., the Borg index) when performing each exercise event that constitutes the target exercise set, and calculate the exercise intensity when the user performs the exercise event based on the perceived intensity of exercise.

The sixth example of exercise intensity estimation (S130) is a combination of multiple of the first to fifth examples above.

The server 30 may estimate the exercise intensity for each section constituting the exercise type, and treat a representative value of the estimated result (e.g., average value, median value, mode value, maximum value, minimum value, first quartile, or third quartile) as the exercise intensity of the exercise type. The client device 10 or the server 30 may identify which section each sensing result is associated with by, for example, analyzing user video data (and user depth data as necessary). Alternatively, in the case where a fixed time is assigned to each section, such as in gymnastics or dance, the client device 10 or the server 30 may identify which section each sensing result is associated with based on the time assignment.
Furthermore, the server 30 may be a combination of two or more of the first to third examples described above.

Incidentally, since the exercise intensity estimated immediately after the start of exercise (for example, within 1 to 2 minutes) is not stable, the estimation of exercise intensity may be omitted for a predetermined period from the start of exercise, or the estimated exercise intensity may be discarded. In other words, the server 30 may estimate the exercise intensity only when it is determined that the exercise intensity is in a plateau state.

After step S130, the server 30 executes determination of the personal index (S131).
Specifically, the server 30 performs a predetermined calculation using the exercise intensity estimated in step S130 to determine the individual index of the exercise intensity for the target event set for the user. As an example, the server 30 treats the exercise intensity estimated for the multiple exercise events constituting the target event set, or the product or ratio of the exercise intensities, as elements (the elements may include values (bias) that are not based on the exercise intensity). The server 30 determines the individual index by multiplying the element by a predetermined coefficient or adding a bias, and then performing a predetermined calculation such as finding the sum, product, or ratio of these. For example, the value obtained by subtracting the exercise intensity of event B from the exercise intensity of event A may be used as the individual index. The calculation formula for the individual index may be defined for each target event set, or may be defined commonly across multiple target event sets.

After step S131, the server 30 executes calculation of individual difference parameters (S132).
Specifically, the server 30 calculates individual difference parameters representing the user's characteristics related to exercise tolerance based on the individual indices determined in step S131 and the reference value of the exercise intensity for the target event set. The server 30 creates a record based on the calculated individual difference parameters and the calculation date (or calculation date and time), and adds the record to the user's parameter log database ( FIG. 8 ).

The reference value of the exercise intensity for the target event set can be determined based on information on the exercise intensity when one or more reference persons (e.g., persons with standard exercise tolerance) perform each of the exercise events constituting the target event set, and on the individual index calculated for each of the persons based on the exercise intensity. Specifically, the exercise intensity when each person performs each exercise event is acquired (actually measured or estimated), an individual index is calculated using the same formula as in step S131, and the reference value can be determined by performing statistical processing on the individual index (e.g., calculation of a representative value such as averaging). The reference value may be calculated in advance for each available target event set and stored in a database. The person referenced to calculate the reference value applied to a user may be determined to have at least one attribute (e.g., age group, sex, or a combination thereof) that is the same as that of the user.

As a first example, the server 30 calculates the individual difference parameter by dividing the individual index by a reference value. In this case, the individual difference parameter corresponds to a prediction result of the ratio of the exercise intensity to be higher or lower than that of a standard person when the user performs the exercise type constituting the target exercise set or another exercise type.

As a second example, the server 30 calculates the individual difference parameter by subtracting a reference value from the personal index. In this case, the individual difference parameter corresponds to a prediction result of how much the exercise intensity will be higher or lower than that of a standard person when the user performs the exercise type that constitutes the target exercise set or another exercise type.

After step S132, the server 30 selects a recommended exercise type (S133).
Specifically, the server 30 refers to the user profile database (FIG. 7) and acquires the user's target intensity information (which is determined based on the results of measuring the user's exercise tolerance as described above). The server 30 selects an exercise type (hereinafter referred to as a "recommended exercise type") that is suitable for the characteristics of the user's exercise tolerance based on the acquired target intensity information, the individual difference parameters calculated in step S132, and the exercise intensity information of each exercise type stored in the exercise type database (FIG. 6). The number of recommended exercise types may be one or more.

As a first example, the server 30 corrects the target value of the user's exercise intensity by the individual difference parameter. For example, the server 30 obtains a corrected target value by dividing the target value by the individual difference parameter or subtracting the individual difference parameter from the target value. Then, the server 30 selects a recommended exercise type from among exercise types whose exercise intensity (hereinafter referred to as "standard intensity") indicated by the corresponding exercise intensity information does not exceed the corrected target value. For example, the server 30 may include in the recommended exercise types an exercise type whose standard intensity is maximum within a range equal to or less than the corrected target value. Note that, when selecting a recommended exercise type, the server 30 may use, instead of the corrected target value, a value obtained by adding or subtracting a margin to the corrected target value, or a value obtained by multiplying the corrected target value by a positive coefficient different from 1. Alternatively, when selecting a recommended exercise type, the server 30 may use, instead of the standard intensity, a value obtained by adding or subtracting a margin to the standard intensity, or a value obtained by multiplying the standard intensity by a positive coefficient different from 1.

As a second example, the server 30 corrects the standard intensity of each exercise type by the individual difference parameter. For example, the server 30 obtains the corrected intensity by multiplying or adding the standard intensity by the individual difference parameter. Then, the server 30 selects the recommended exercise type from the exercise types whose corrected intensity does not exceed the target value of the exercise intensity of the user. For example, the server 30 may include in the recommended exercise types the exercise types whose corrected intensity is maximum within a range below the target value. Note that, when selecting the recommended exercise type, the server 30 may use, instead of the target value, a value obtained by adding or subtracting a margin to the target value, or a value obtained by multiplying the target value by a positive coefficient different from 1. Alternatively, when selecting the recommended exercise type, the server 30 may use, instead of the corrected intensity, a value obtained by adding or subtracting a margin to the corrected intensity, or a value obtained by multiplying the corrected intensity by a positive coefficient different from 1.

After step S133, the server 30 transmits recommended exercise event information (S134).
Specifically, the server 30 transmits information about the recommended exercise selected in step S133 (hereinafter referred to as "recommended exercise information") to the client device 10. The recommended exercise information may be, for example, information that identifies the recommended exercise, or may be screen information for displaying the recommended exercise.

After step S134, the client device 10 executes screen display (S113).
Specifically, the client device 10 receives the recommended exercise information transmitted by the server in step S134. The client device 10 displays on the display 15 a screen based on the recommended exercise information.

For example, the client device 10 displays the screen of FIG. 10 on the display 15. The screen of FIG. 10 includes objects J20 to J23.

The object J20 displays the recommended exercise type. The object J20 also accepts a user instruction to start the recommended exercise type or to play a model video of the recommended exercise type. When the object J20 is selected, the client device 10 may re-execute the exercise type recommendation process of this embodiment with the exercise type corresponding to the object J20 as an exercise type that constitutes a new target exercise set. Here, the server 30 may calculate the user's individual index for the new target exercise set, and calculate the user's individual difference parameters based on the individual index and a reference value corresponding to the new target exercise set. Alternatively, when the object J20 is selected, the client device 10 may play a model video of the exercise type corresponding to the object J20.

Object J21 accepts a user instruction to select an exercise type from among those other than the recommended exercise types. When object J21 is selected, the client device 10 may, for example, display a list of exercise types other than the recommended exercise types and accept a user instruction to select an exercise type. In response to the acceptance of such a user instruction, the client device 10 may re-execute the exercise type recommendation process of this embodiment with the selected exercise type as an exercise type that constitutes a new target exercise set. Here, the server 30 may calculate the user's individual index for the new target exercise set, and calculate the user's individual difference parameters based on the individual index and a reference value corresponding to the new target exercise set. Alternatively, when any exercise type is selected, the client device 10 may play a model video of the exercise type.

Object J22 accepts a user instruction to end the exercise. When object J22 is selected, the client device 10 ends the exercise type recommendation process of this embodiment.

After step S113, the client device 10 may terminate the exercise type recommendation process (Figure 9).

(4-2) Parameter Monitoring Processing The parameter monitoring processing of this embodiment will be described with reference to FIG.

The parameter monitoring process of this embodiment may be started, for example, each time the exercise type recommendation process (FIG. 9) of this embodiment (particularly, the calculation of individual difference parameters (S133)) is performed, or may be repeatedly performed at a predetermined interval for each user.

As shown in FIG. 11, the server 30 acquires parameter log information (S230).
Specifically, the server 30 refers to the parameter log database of the user to be processed and acquires the parameter log information.

After step S230, the server 30 executes a determination of a predetermined condition (S231).
Specifically, the server 30 determines whether or not a predetermined condition is satisfied for the parameter log information acquired in step S230. For example, the predetermined condition may include at least one of the following.
The representative value of a certain number of recent individual difference parameters (which may be one or more) exceeds a threshold value (i.e., the user is predicted to have an excessively high exercise intensity compared to a typical person).
The representative value of the individual difference parameter over the most recent predetermined period exceeds a threshold value (i.e., the user is predicted to exercise at an excessively high intensity compared to a typical person).
The rate of increase in the individual difference parameter for a given number of recent times exceeds a threshold value (i.e., it is predicted that the user's exercise intensity for the same exercise type is rapidly increasing)
The rate of increase in the individual difference parameter in the most recent specified period exceeds a threshold value (i.e., the user's exercise intensity for the same exercise type is predicted to increase rapidly).

If it is determined in step S231 that the predetermined condition is met, the server 30 executes output of an alert (S232).
Specifically, the server 30 transmits the alert to the user's client device 10 or a predetermined terminal. The predetermined terminal may be a terminal of a person designated by the user (e.g., a family member), a terminal of the user's doctor, or a terminal of a person who plans or instructs the user's exercise therapy.

The alert may include, for example, at least one of the following information:
A recent predetermined number of individual difference parameters A recent individual difference parameter over a recent predetermined period of time An increase rate of a recent predetermined number of individual difference parameters An increase rate of an individual difference parameter over a recent predetermined period of time A message (e.g., text, image, sound, or a combination thereof) indicating that the user is suspected of having an exacerbation of a heart disease
A message indicating that the user is suspected of having a physical injury A message indicating that the user is suspected of having a temporary deterioration in health

The alert output destination (user's client device 10 or a specified terminal) presents information based on the alert received from the server 30 in the form of an image, sound, vibration, or a combination of these.

After step S232, the server 30 terminates the parameter monitoring process of this embodiment.

If it is determined in step S231 that the specified condition is not met, the server 30 skips outputting the alert (S232) and terminates the parameter monitoring process of this embodiment.

(5) Summary As described above, the server 30 of this embodiment acquires first data based on the sensing results when the user is performing a first type of exercise, and acquires a first exercise intensity when the user is performing the first type of exercise based on the first data. The server 30 acquires second data based on the sensing results when the user is performing a second type of exercise different from the first type of exercise, and acquires a second exercise intensity when the user is performing the second type of exercise based on the second data. The server 30 determines a first individual index of the user by performing a predetermined calculation on multiple exercise intensities acquired for the user, including the first exercise intensity and the second exercise intensity, and calculates a parameter representing the user's characteristics related to exercise tolerance based on the first individual index of the user and the first reference value. This makes it possible to use the individual difference parameter as a decision material for planning or instructing an exercise therapy provided to the user (which may include a prescription), or as a signal indicating the user's physical function or physical condition. In addition, by referring to the exercise intensities for multiple types of exercise, a more reliable individual index can be calculated.

The server 30 may use the parameters, the user's target exercise intensity, and a database that associates each of the multiple exercise types with a standard exercise intensity to select a recommended exercise type for the user from multiple exercise types, and output information indicating the recommended exercise type. This allows the server 30 to recommend an exercise type not only by matching the measurement results of the user's exercise tolerance with the standard exercise intensity of each exercise type, but also by taking into consideration the characteristics of the user's exercise tolerance.

The server 30 may select, as a recommended exercise type, an exercise type associated with an exercise intensity that does not exceed a corrected exercise intensity obtained by correcting the target exercise intensity using a parameter. This makes it possible to recommend an exercise type whose exercise intensity is expected not to exceed the target exercise intensity when actually performed by the user.

The server 30 may select recommended exercise types from among a plurality of exercise types so as to include an exercise type associated with a maximum exercise intensity that does not exceed the corrected exercise intensity. This makes it possible to recommend an exercise type that is estimated to have an exercise intensity that does not exceed the target exercise intensity when actually performed by the user and is close to the target exercise intensity.

The target exercise intensity may be determined according to the exercise intensity specified by an algorithm or by a person who plans or instructs the user's exercise therapy based on the measurement results of the user's exercise tolerance. This makes it possible to recommend an exercise type that matches the exercise intensity that is determined to be suitable for the user from a medical perspective.

The first reference value may be a representative value of the first individual indexes of multiple people calculated using the same formula as the first individual index of the user. This makes it possible to obtain an individual difference parameter that appropriately represents the characteristics of the user's exercise tolerance.

The multiple people may be defined to have at least one attribute that is the same as the user. This allows for the acquisition of individual difference parameters that more appropriately represent the characteristics of the user's exercise tolerance.

The multiple exercise types may include exercise types that can be performed without using a device that can adjust the exercise load. This can enhance the range of exercise types available to the user.

The first exercise type may have a different standard exercise intensity from the second exercise type. This makes it easier to obtain appropriate individual difference parameters, for example, when calculating an individual index using the difference or ratio between the exercise intensities.

The multiple exercise intensities may be represented using at least one of oxygen consumption or energy consumption. This allows for obtaining individual difference parameters that represent the characteristics of the user's oxygen consumption or energy consumption.

The server 30 may acquire third data based on the sensing results when the user is performing a third exercise type selected from the recommended exercise types, and acquire a third exercise intensity when the user is performing the third exercise type based on the third data. The server 30 may determine a second individual index of the user by performing a predetermined calculation on multiple exercise intensities acquired for the user, including the third exercise intensity, and calculate a parameter based on the second individual index and a second reference value. This makes it possible to calculate individual difference parameters for various combinations of exercise types.

(6) Estimation Model As described above, the server 30 may estimate the exercise intensity using an estimation model. In this case, the estimation model corresponds to a trained model created by supervised learning using a teacher data set described below, or a derived model or distilled model of the trained model. The estimation model may be constructed for each type of exercise, or may be constructed commonly across multiple types of exercise.

(6-1) Teacher Data Set A teacher data set that can be used in supervised learning for constructing an estimation model will be described below. Fig. 12 is a diagram showing the data structure of a teacher data set that can be used in this embodiment.

As shown in FIG. 12, the teacher dataset includes multiple teacher data. The teacher data is used to train or evaluate the model to be learned (hereinafter referred to as the "target model"). The teacher data includes a sample ID, input data, and correct answer data.

The sample ID is information that identifies the teacher data.

The input data is data that is input to the target model during training or evaluation. The input data corresponds to examples used during training or evaluation of the target model. As an example, the input data is data regarding the subject's physical state during exercise (i.e., relatively dynamic data) and data regarding the subject's health condition (i.e., relatively static data). At least a portion of the data regarding the subject's physical state is obtained by analyzing the subject's physical state with reference to the subject video data (or the subject video data and subject depth data).

The subject video data is data related to a subject video showing the subject exercising. The subject video data can be obtained, for example, by capturing a picture of the subject's appearance (e.g., the whole body) from the front or diagonally in front (e.g., 45 degrees forward) with a camera (e.g., a camera mounted on a smartphone) during an examination related to exhaled gas (e.g., a CPX examination).

Subject depth data is data regarding the distance (depth) from the depth sensor to each part of the subject during exercise. Subject depth data can be obtained by operating the depth sensor when taking video of the subject.

The subject may be the same person as the user for whom estimation of exercise intensity based on exercise tolerance is performed when the information processing system 1 is operated, or may be a different person. By making the subject and the user the same person, the target model may learn the personality of the user, which may improve the estimation accuracy. On the other hand, allowing the subject to be a different person from the user has the advantage that it is easier to enrich the teacher dataset. In addition, the subjects may be composed of multiple people including the user, or multiple people excluding the user.

In the example of Figure 12, the input data includes skeletal data, facial expression data, skin color data, breathing data, heart rate data, and health condition data.

Skeletal data is data (e.g., features) related to the skeleton of a subject during exercise. Skeletal data includes, for example, data related to the speed or acceleration of each part of the subject (which may include data related to changes in the parts of the muscles used by the subject, or data related to the subject's bodily sensation). Skeletal data can be obtained by analyzing the skeleton of a subject during exercise with reference to subject video data (or subject video data and subject depth data). As an example, Vision, an SDK for iOS 14, or other skeleton detection algorithms (e.g., OpenPose, PoseNet, MediaPipe Pose) can be used to analyze the skeleton. Alternatively, skeletal data for a training dataset can be obtained, for example, by having a subject exercise with motion sensors attached to each part of the body.

The results of skeletal detection can be used for quantitative evaluation, qualitative evaluation, or a combination of both of these of exercise. As a first example, the results of skeletal detection can be used to count the number of reps. As a second example, the results of skeletal detection can be used to evaluate the form of an exercise, or the appropriateness of the load applied by the exercise. For example, if the exercise is squats, the results of skeletal detection can be used to evaluate whether the knees are sticking out too far forward, resulting in a dangerous form, or whether the hips are lowered firmly and deeply to apply sufficient load.

Facial expression data is data (e.g., features) about a subject's facial expressions while exercising. Facial expression data can be analyzed by applying an algorithm or trained model to the subject's video data. Alternatively, facial expression data for a training dataset can be obtained, for example, by a human labeling the subject's videos.

Skin color data is data (e.g., features) about the skin color of a subject during exercise. Skin color data can be analyzed by applying an algorithm or trained model to the subject video data. Alternatively, skin color data for a training dataset can be obtained, for example, by a human labeling the subject video.

The respiratory data is data (e.g., features) related to the subject's breathing during exercise. The respiratory data relates, for example, to the number of breaths per unit time or to a breathing pattern. The breathing pattern may include at least one of the following:
Ventilation rate, ventilation volume, and ventilation speed (i.e., ventilation volume per unit time, or ventilation rate)
- Ventilation acceleration (i.e., the time derivative of the ventilation rate)
・Carbon dioxide emission concentration ・Carbon dioxide emission volume (VCO2)
・Oxygen intake concentration ・Oxygen intake volume (VO2)
The data relating to the breathing pattern may include data that can be calculated based on a combination of the above data, such as the gas exchange ratio R (=VCO2/VO2).

The respiration data can be obtained, for example, by analyzing the skeletal data. As an example, the following items can be analyzed from the skeletal data:
Movement (spreading) of the shoulders, chest (which may include the lateral chest), abdomen, or a combination thereof
Inhalation time, exhalation time, and use of accessory respiratory muscles

The respiratory data for the teacher dataset can be obtained, for example, from the results of a test on exhaled gas performed on a subject while exercising. Details of the exhaled gas tests that can be performed on a subject while exercising will be described later. Alternatively, the ventilation rate, ventilation volume, ventilation speed, or ventilation acceleration of the respiratory data for the teacher dataset can be obtained, for example, from the results of a respiratory function test (e.g., a pulmonary function test or a spirometry test) performed on a subject while exercising. In this case, the respiratory function test can be performed using commercially available testing equipment rather than medical equipment.

Heart rate data is data (e.g., features) related to the subject's heart rate during exercise. Heart rate data can be obtained, for example, by analyzing subject video data or the analysis results thereof (e.g., skin color data). Alternatively, heart rate data for the teacher dataset may be obtained, for example, from the results of a test on exhaled gas together with respiratory data, which will be described later. Subject heart rate data for the teacher dataset can also be obtained by having the subject perform the above exercise while wearing a heart rate sensor or electrodes for an electrocardiogram monitor.

Health condition data is data related to the subject's health condition. Health condition data can be obtained in various ways. The subject's health condition data may be obtained before, during, or after the subject's exercise. The subject's health condition data may be obtained based on a report from the subject or the subject's doctor, may be obtained by extracting information linked to the subject in a medical information system, or may be obtained via the subject's app (e.g., a healthcare app).

The health condition includes at least one of the following:
Age, sex, height, weight, body fat percentage, muscle mass, bone density, current illness history, past medical history, medication history, surgical history, lifestyle history (e.g., smoking history, drinking history, activities of daily living (ADL), frailty score, etc.)
・Family history ・Results of respiratory function tests ・Results of tests other than respiratory function tests (for example, results of blood tests, urine tests, electrocardiograms (including Holter ECGs), cardiac ultrasounds, X-rays, CT scans (including cardiac morphological CT and coronary artery CT), MRI scans, nuclear medicine tests, PET scans, etc.)
Data obtained during cardiac rehabilitation (including Borg index)

The correct answer data is data that corresponds to the correct answer for the corresponding input data (example question). The target model is trained (supervised learning) to output the correct answer data for the input data as close as possible to the correct answer data. As an example, the correct answer data represents exercise intensity.

Exercise intensity is an index for quantitatively evaluating the exercise load imposed on a subject. Exercise intensity can be expressed numerically using at least one of the following:
・Energy (calorie) consumption ・Oxygen consumption ・Heart rate

The correct answer data can be obtained, for example, from the results of a breath gas test performed on a subject during exercise. A first example of a breath gas test is a test (typically a CPX test) performed while a subject wearing a breath gas analyzer is performing incremental exercise (e.g., an ergometer). A second example of a breath gas test is a test performed while a subject wearing a breath gas analyzer is performing exercise of a constant or variable intensity (e.g., bodyweight exercise, gymnastics, strength training).

Alternatively, correct answer data can be obtained from the results of tests other than exhaled gas administered to a subject during exercise. Specifically, correct answer data can be obtained from the results of a cardiopulmonary exercise load prediction test based on measuring the lactate concentration in the sweat or blood of a subject during exercise. A wearable lactate sensor can be used to measure the lactate concentration of the subject.

It is also possible to construct an estimation model for each of a plurality of health condition categories based on (at least a part of) the subject's health condition. In this case, (at least a part of) the user's health condition may be referenced to select the estimation model. In this further variation, the input data for the estimation model may be data that is not based on the user's health condition, or may be data that is based on the user's health condition and user video.

(7) Other Modifications The storage device 11 may be connected to the client device 10 via the network NW. Each input device or output device may be integrated with the client device 10. The storage device 31 may be connected to the server 30 via the network NW. Each input device or output device may be integrated with the wearable device 50.

An example has been shown in which the information processing system 1 of the embodiment is implemented by a client/server type system. However, the information processing system 1 of the embodiment can also be implemented by a peer-to-peer type system or a standalone computer. As an example, the client device 10 may estimate the exercise intensity.

Each step of the above information processing can be executed by either the client device 10 or the server 30. As an example, the server 30, instead of the client device 10, may obtain at least a portion of the user data or the user skeletal data by analyzing the user video (or the user video and the user depth).

One or more steps of the above information processing may be performed using a trained model.

In the above example, a recommended exercise type is selected based on the standard exercise intensity of each of a plurality of exercise types, the user's individual difference parameters, and the results of measuring the exercise tolerance of the user. However, the server 30 may determine instructions for adjusting the exercise intensity for the exercise type selected by the user based on these pieces of information (in other words, presenting an exercise type (variation) in which only a part of the exercise definition information is different from the exercise type selected by the user), and output the information of the instructions to the client device 10. The instructions may include, for example, the form of the exercise type (for example, the positional relationship or angle of a part (which may include a part that is not moved during exercise), or the range of motion of a part that is moved during exercise), pace, number of reps, or the time or number of rest periods. As a result, for example, even if the exercise intensity of the exercise type selected by the user is estimated to exceed the above-mentioned target value, it is possible to prevent the user's exercise intensity from deviating from the target value by specifying a form with a lighter exercise intensity than the standard, slowing down the pace than the standard, reducing the number of reps than the standard, or increasing the time or number of rest periods. Alternatively, even if the exercise intensity of the user's selected exercise type is estimated to be below the target value, the user can prevent the exercise intensity from deviating from the target value by specifying a form that results in a heavier exercise intensity than standard, setting a faster pace than standard, increasing the number of repetitions than standard, or reducing the time or number of rest periods.

In the above description, an example was shown in which a recommended type of exercise was selected based on the standard exercise intensity of each of a plurality of types of exercise, the user's individual difference parameters, and the results of measuring the exercise tolerance of the user. However, the server 30 may select a recommended type of exercise based on at least one of the pulse rate, respiration rate, or perceived exercise intensity of the user during exercise. Here, the perceived exercise intensity can be acquired by, for example, receiving an input of the Borg index from the user via the client device 10 during a rest period. When at least one of the pulse rate, respiration rate, or perceived exercise intensity exceeds an upper limit, the server 30 may select an exercise type with a lower standard exercise intensity as a recommended type of exercise compared to cases where the pulse rate, respiration rate, or perceived exercise intensity is below a lower limit. However, when an upper limit of exercise intensity is set for the user by a doctor, an exercise type corresponding to an exercise intensity exceeding the upper limit is not selected.
Furthermore, the pulse rate, respiration rate, or perceived exertion intensity of the user during exercise may be used to adjust the exercise intensity of the next set when performing an exercise event consisting of multiple sets. For example, when at least one of the pulse rate, respiration rate, or perceived exertion intensity exceeds an upper limit, the server 30 may determine instructions (e.g., form, pace, number of reps, or time or number of rest periods) so that the exercise intensity of the next set is lower than that of the current set, and output information on the instructions to the client device 10. On the other hand, when at least one of the pulse rate, respiration rate, or perceived exertion intensity falls below a lower limit, the server 30 may determine instructions so that the exercise intensity of the next set is higher than that of the current set, and output information on the instructions to the client device 10. However, when an upper limit of exercise intensity is set for the user by a doctor, instructions corresponding to an exercise intensity exceeding the upper limit are not selected.
For example, the server 30 may determine a recommended type of exercise or instructions so that the exercise intensity of the next set is reduced by 0.2 METs when the average heart rate is 5 or more higher than the target heart rate or the perceived intensity of exercise is 14 or higher. The server 30 may determine a recommended type of exercise or instructions so that the exercise intensity of the next set is reduced by 0.4 METs when the average heart rate is 10 or more higher than the target heart rate or the perceived intensity of exercise is 16 or higher. The server 30 may also determine a recommended type of exercise or instructions so that the exercise intensity of the next set is increased by 0.2 METs when the average heart rate is 5 or more lower than the target heart rate and the perceived intensity of exercise is less than 10.

The client device 10 may further execute the following processes, for example, when acquiring the sensing data (S110) (in other words, while the user is exercising). Specifically, the client device 10 estimates the user's movements during exercise (e.g., movements of the skeleton or other feature points over multiple points of time, or the state of the skeleton or other feature points at a single point of time) based on the sensing data. Then, when the result of the estimation of the movements does not match at least one of the form or pace determined for the type of exercise being performed by the user (e.g., the range of motion of a part is too narrow or too wide, the angle of a part deviates from the standard, or the pace is too fast or too slow, etc.), the client device 10 provides feedback to the user. The feedback may include at least one of the following:
- Output of light, sound or voice (e.g., voice informing the user of what they have done incorrectly) - Display of an image (e.g., an image informing the user of what they have done incorrectly) - Vibration of the wearable device This allows for a more accurate estimation of the user's personal indicators.
Alternatively, when the number or frequency of occurrences of the estimation result relating to the movement not conforming to at least one of the form or pace defined for the type of exercise being performed by the user exceeds a threshold, the client device 10 may recommend to the user to repeat the same type of exercise or to perform a different type of exercise. In this case, the client device 10 may omit the processes subsequent to the generation of the user data (S111).

Instead of the exercise type recommendation process, the information processing system 1 may perform a process for measuring the oxygen consumption at the anaerobic metabolic threshold. This process may be automatically selected, for example, during the first three days after the user starts the exercise therapy, or may be selected by the user or the person who plans or instructs the exercise therapy. In this mode, the information processing system 1 sequentially selects exercise types so that the exercise intensity increases by 0.2 METs until either of the following conditions is met: the average heart rate is 5 or more higher than the target heart rate, or the subjective intensity of exercise is 14 or more, and the user exercises. The information processing system 1 treats the oxygen consumption estimated for the exercise type performed immediately before the exercise type for which the above condition is met as the oxygen consumption at the user's anaerobic metabolic threshold. The information processing system 1 may also treat the exercise intensity corresponding to the exercise type for which the average heart rate closest to the target heart rate is obtained as the optimal exercise intensity.

In the above description, an example was given in which an alert is output when the user's individual difference parameters satisfy a predetermined condition. However, the alert may be output by various triggers other than the individual difference parameters. As an example, the server 30 accumulates data on the types of exercise the user has performed in the past and the user's heart rate, perceived intensity of exercise, or respiration rate when performing the type of exercise. The server 30 may then output an alert when the user's heart rate, perceived intensity of exercise, or respiration rate during exercise exceeds a threshold value and increases (worsens) compared to the user's heart rate, perceived intensity of exercise, or respiration rate when performing the same type of exercise in the past. In addition, if the degree of distress of the user (hereinafter referred to as "distress degree") is quantified based on, for example, facial expression data or skeletal data, it can be used as a trigger similar to the above-mentioned heart rate, perceived intensity of exercise, or respiration rate. In addition, the server 30 may use as a trigger an input regarding the emergence or worsening of the user's subjective symptoms of weight gain, shortness of breath, edema, fatigue, loss of appetite, or insomnia in daily life, or a sensor input suggesting the emergence or worsening of these symptoms.

In the above description, an example of calculating the user's individual difference parameters in the exercise type recommendation process or parameter monitoring process has been shown. However, the user's individual difference parameters may be calculated based on data obtained during a CPX test, for example. For example, the individual difference parameters may be calculated based on the exercise intensity (actual measured value of oxygen intake, for example) when the user is rowing the ergometer with a first load (for example, 20 W) and the exercise intensity when the user is rowing the ergometer with a second load (for example, 40 W). When a doctor sets an upper limit of exercise intensity for a user (an example of exercise prescription), the exercise type to be presented to the doctor can be selected based on the CPX data and the individual difference parameters. As an example, when 3.6 METs is recommended as the upper limit based on CPX, a group of display areas of sample videos of exercise types with the above-mentioned corrected intensity corresponding to 3.4 METs, a group of display areas of sample videos of exercise types with the corrected intensity corresponding to 3.6 METs, and a group of display areas of sample videos of exercise types with the corrected intensity corresponding to 3.8 METs may be arranged on the UI screen. Additionally, information about exercise events (e.g., sample videos) may be sorted in units smaller than 0.2 METs or larger.

In the above embodiment, an example was shown in which a user's individual difference parameters were calculated based on the exercise intensity when the user was performing a number of different exercise types. However, instead of one of the exercise types, the user's exercise intensity in a resting state may be used. By defining a standard exercise intensity in a resting state and acquiring the user's exercise intensity in a resting state, the individual difference parameters can be calculated in a similar manner.

In the above description, an example is shown in which a user video is captured using the camera 16 of the client device 10. However, the user video may be captured using a camera other than the camera 16. An example is shown in which the user depth is measured using the depth sensor 17 of the client device 10. However, the user depth may be measured using a depth sensor other than the depth sensor 17.

In the above description, an example has been given in which the wearable device 50 measures the user's heart rate. However, the heart rate can be obtained by analyzing (e.g., rPPG (Remote Photoplethysmography) analysis) video data or the analysis results thereof (e.g., skin color data). The analysis of the heart rate may be performed by a trained model constructed using machine learning technology. Alternatively, the user may be made to exercise while wearing electrodes for an electrocardiogram monitor, so that the electrocardiogram monitor can measure the user's heart rate. In these modified examples, the user does not need to wear the wearable device 50 to measure the heart rate.

The wearable device 50 may be equipped with a sensor for measuring at least one of the following items instead of or in addition to the heart rate sensor 56 and the acceleration sensor 57.
Blood glucose level and oxygen saturation The measurement results of each sensor may be used as appropriate in generating input data, estimating exercise intensity or ventilation index, presenting information based on the estimation results, or in other situations. As an example, the measurement results of blood glucose level may be referenced to evaluate exercise intensity converted into energy consumption or oxygen consumption. As another example, the measurement results of acceleration may be used to determine the score of a user's exercise (e.g., gymnastics).

Acceleration data can also be used as part of the input data for the estimation model described in this embodiment or the modified example. Alternatively, the user's skeleton can be analyzed by referring to the acceleration data. The acceleration data can be acquired, for example, by acceleration sensor 19 or acceleration sensor 57 when shooting a user video.

It is also possible to use oxygen saturation data as part of the input data for the estimation model described in this embodiment or the modified example. The oxygen saturation data can be obtained, for example, by having the user wear a wearable device equipped with a sensor (e.g., an optical sensor) capable of measuring blood oxygen concentration when shooting a user video, or a pulse oximeter. The oxygen saturation data may be estimated, for example, by performing rPPG analysis on the user video data.

The information processing system 1 of the present embodiment and each modified example is also applicable to a video game in which the game progress is controlled according to the player's body movements. The video game may be a mini-game that can be played while the above-mentioned treatment app, rehabilitation app, or fitness app is being executed, or may be configured as a standalone app. As an example, the information processing system 1 may estimate the user's exercise intensity during game play, and determine one of the following depending on the result of the estimation (e.g., a numerical value indicating the user's exercise intensity). This can enhance the effect of the video game on improving the user's health.
The quality (e.g., difficulty) or quantity of video game challenges (e.g., stages, missions, quests) given to the user. The quality (e.g., type) or quantity of video game rewards (e.g., in-game currency, items, bonuses) given to the user. Game parameters related to the progress of the video game (e.g., score, damage).

The information processing system 1 of the present embodiment and each modified example is also applicable to a video game in which the game progress is controlled according to the movement of the player's body. The video game may be a mini-game that can be played during the execution of the above-mentioned treatment app, rehabilitation app, or fitness app, or may be configured as a standalone app. As an example, the information processing system 1 performs an estimation of the user's skeleton based on the user's video during game play. The estimation of the user's skeleton may be performed based on at least one of the user depth or the user acceleration in addition to the user's video. The information processing system 1 evaluates the degree to which the user's posture during exercise (e.g., gymnastics) matches an ideal posture (model) based on the result of the estimation of the user's skeleton. The information processing system 1 may determine one of the following depending on the result of this evaluation (e.g., a numerical value indicating the degree to which the user's posture matches the ideal posture). This can enhance the effect of the video game on improving the user's health.
The quality (e.g., difficulty) or quantity of video game challenges (e.g., stages, missions, quests) given to the user. The quality (e.g., type) or quantity of video game rewards (e.g., in-game currency, items, bonuses) given to the user. Game parameters related to the progress of the video game (e.g., score, damage).

In addition to or instead of the microphone 18, a microphone of the wearable device 50 (a microphone provided in the wearable device 50 or connected to the wearable device 50) may receive sound waves emitted by the user when capturing a user video and generate sound data. The sound data may constitute input data for the estimation model described in this embodiment or the modified example. The sound emitted by the user is, for example, at least one of the following:
- Sound waves generated by the rotation of the user's legs (e.g., sounds generated by the pedals or the actuators connected to the pedals)
- Sounds generated by the user's breathing or speaking

In the above explanation, a CPX test is given as an example of a test related to exhaled gas. In a CPX test, a gradually increasing exercise load is applied to the test subject. However, it is not necessary to gradually increase the exercise load applied to the user when shooting the user video. Specifically, the real-time exercise intensity can be estimated even when the user is given a constant or a constantly changeable exercise load. For example, the exercise performed by the user may be bodyweight exercise, calisthenics, or strength training.

Although the embodiments of the present invention have been described in detail above, the scope of the present invention is not limited to the above-mentioned embodiments. Furthermore, the above-mentioned embodiments can be improved or modified in various ways without departing from the spirit of the present invention. Furthermore, the above-mentioned embodiments and modifications can be combined.

1: Information processing system 10: Client device 11: Storage device 12: Processor 13: Input/output interface 14: Communication interface 15: Display 16: Camera 17: Depth sensor 18: Microphone 19: Acceleration sensor 21: Display 30: Server 31: Storage device 32: Processor 33: Input/output interface 34: Communication interface 50: Wearable device 51: Storage device 52: Processor 53: Input/output interface 54: Communication interface 55: Display 56: Heart rate sensor 57: Acceleration sensor

Claims (15)

Computer,
A means for acquiring first data based on a sensing result when a user is performing a first type of exercise;
a means for acquiring a first exercise intensity when the user is performing the first exercise event based on the first data;
A means for acquiring second data based on sensing while the user is performing a second exercise type different from the first exercise type or while the user is at rest;
A means for acquiring a second exercise intensity when the user is performing the second exercise type or at rest based on the second data;
means for determining a first personal index of the user by performing a predetermined calculation on a plurality of exercise intensities acquired for the user, including the first exercise intensity and the second exercise intensity;
A means for calculating a parameter representing a characteristic of the user related to exercise tolerance based on a first individual index of the user and a first reference value of exercise intensity for a target event set which may include a resting state ;
A program that functions as a The computer,
a means for selecting a recommended exercise type for the user from among the plurality of exercise types using the parameters, a target exercise intensity for the user, and a database that associates each of a plurality of exercise types with a standard exercise intensity;
A means for outputting information indicating the recommended exercise type;
To function as
The program according to claim 1. the selecting means selects, as the recommended exercise type, an exercise type associated with an exercise intensity not exceeding a corrected exercise intensity obtained by correcting the target exercise intensity using the parameter;
The program according to claim 2. The selecting means selects the recommended exercise type so as to include an exercise type associated with a maximum exercise intensity not exceeding the corrected exercise intensity among the plurality of exercise types.
The program according to claim 3. The target exercise intensity is determined according to an exercise intensity designated by a person who plans or instructs the exercise therapy of the user or an algorithm based on the measurement result of the exercise tolerance of the user.
The program according to claim 2. the first reference value is a representative value of first individual indicators of a plurality of persons calculated using the same calculation formula as the first individual indicator of the user;
The program according to claim 1. The plurality of persons are defined to have at least one attribute identical to that of the user.
The program according to claim 6. The computer,
means for making an estimation regarding a movement of the user when performing the first type of exercise based on the first data;
means for providing feedback to the user when the result of the estimation regarding the movement does not conform to at least one of a form or a pace defined for the first athletic event;
The program according to claim 1 , The plurality of exercise events include an exercise event that can be performed without using a device capable of adjusting the exercise load,
The program according to claim 2. The first exercise event has a different standard exercise intensity from the second exercise event.
The program according to claim 1. The plurality of exercise intensities are expressed using at least one of oxygen consumption or energy consumption.
The program according to claim 1. The computer,
a means for acquiring third data based on a sensing result when the user is performing a third exercise selected from the recommended exercises;
a means for acquiring a third exercise intensity when the user is performing the third exercise event based on the third data;
means for determining a second personal index of the user by performing a predetermined calculation on a plurality of exercise intensities acquired for the user, including the third exercise intensity;
A means for calculating the parameters based on the second individual index and a second reference value of exercise intensity for a target event set, which may include a resting state ;
To function as
The program according to claim 2. A means for acquiring first data based on a sensing result when a user is performing a first type of exercise;
A means for acquiring a first exercise intensity when the user is performing the first exercise event based on the first data;
A means for acquiring second data based on sensing while the user is performing a second exercise type different from the first exercise type or while the user is at rest;
A means for acquiring a second exercise intensity when the user is performing the second exercise type or at rest based on the second data;
a means for determining a first individual index of the user by performing a predetermined calculation on a plurality of exercise intensities acquired for the user, the plurality of exercise intensities including the first exercise intensity and the second exercise intensity;
and means for calculating a parameter representing a characteristic of the user related to exercise tolerance based on a first individual index of the user and a first reference value of exercise intensity for a target event set that may include a time at rest . The computer
acquiring first data based on a sensing result when a user is performing a first exercise event;
acquiring a first exercise intensity when the user is performing the first exercise event based on the first data;
acquiring second data based on sensing while the user is performing a second exercise type different from the first exercise type or while the user is at rest;
acquiring a second exercise intensity when the user is performing the second exercise type or at rest based on the second data;
determining a first personal index of the user by performing a predetermined calculation on a plurality of exercise intensities acquired for the user, the plurality of exercise intensities including the first exercise intensity and the second exercise intensity;
and calculating a parameter representing the user's characteristics related to exercise tolerance based on a first personal index of the user and a first reference value of exercise intensity for a target event set that may include a resting state . A system including a plurality of information processing devices,
A means for acquiring first data based on a sensing result when a user is performing a first type of exercise;
A means for acquiring a first exercise intensity when the user is performing the first exercise event based on the first data;
A means for acquiring second data based on sensing while the user is performing a second exercise type different from the first exercise type or while the user is at rest;
A means for acquiring a second exercise intensity when the user is performing the second exercise type or at rest based on the second data;
a means for determining a first individual index of the user by performing a predetermined calculation on a plurality of exercise intensities acquired for the user, the plurality of exercise intensities including the first exercise intensity and the second exercise intensity;
and means for calculating a parameter representing a characteristic of the user relating to exercise tolerance based on a first personal index of the user and a first reference value of exercise intensity for a set of target events, which may include a time at rest .