JP7482471B2 - How to generate a learning model - Google Patents

Description

The present invention relates to a method and device for generating a learning model for analyzing the motion of an object, and a motion analysis device, method, and program for analyzing the motion of an object using the learning model.

Conventionally, devices that capture a golf swing with a camera and analyze the golf swing based on the captured image have been known (Patent Document 1, etc.). The results of the analysis are used for various purposes, such as fitting golf clubs to suit golfers, improving a golfer's form, and developing golf equipment. In recent years, distance image sensors that can measure depth images have also become widespread. Patent Document 1 discloses that depth images captured by a distance image sensor are input into a learning model consisting of a neural network, and a motion value that represents the motion of an object is derived from the learning model.

JP 2018-26131 A

When analysis is performed using a learning model, as in Patent Document 1, the accuracy of the analysis depends on whether an accurate learning model can be constructed. Furthermore, whether an accurate learning model can be constructed depends on the training data used for training. In other words, if the amount of training data is small or only training data of insufficient quality is collected, an accurate learning model cannot be obtained. This applies not only to the golf swing analysis example given at the beginning, but also to all situations in which the movement of an object is analyzed.

The present invention aims to provide a method and device for easily generating a highly accurate learning model, as well as a motion analysis device, method, and program for analyzing the motion of an object using the learning model.

A method for generating a learning model according to a first aspect is a method for generating a learning model for analyzing the movement of an object, and includes the following (1) to (5).
(1) Obtaining bone data representing the skeleton of the object; (2) Creating three-dimensional computer graphics (CG) data representing the body shape of the object based on the bone data; (3) Creating a number of depth images of the object based on the three-dimensional CG data; (4) Obtaining a number of motion values representing the motion of one or more specific parts of the object, each corresponding to the number of depth images; (5) Using the depth images based on the three-dimensional CG data and a number of data sets of the corresponding motion values as training data, training a neural network that uses depth images of the object captured by a range image sensor as input and the motion values as output.

The method for generating a learning model according to the second aspect is the method for generating a learning model according to the first aspect, and (4) includes deriving the multiple motion values based on at least one of the bone data and the three-dimensional CG data.

The method for generating a learning model according to the third aspect is a method for generating a learning model according to the first or second aspect, and (2) includes creating the three-dimensional CG data representing different body types by fleshing out the same skeleton in different ways.

The method for generating a learning model according to the fourth aspect is a method for generating a learning model according to any one of the first aspect to the third aspect, and (3) includes creating multiple depth images with different viewpoints based on the 3D CG data of the same scene.

A method for generating a learning model according to a fifth aspect is a method for generating a learning model according to any one of the first aspect to the fourth aspect, and further includes the following (6) to (8).
(6) acquiring a number of depth images of the object photographed by the distance image sensor; (7) acquiring a number of the motion values corresponding to the number of depth images photographed by the distance image sensor; (8) training the neural network using the number of data sets of the depth images based on the 3D CG data and the corresponding motion values as training data, and then training the neural network using the number of data sets of the depth images photographed by the distance image sensor and the corresponding motion values as training data.

The method for generating a learning model according to the sixth aspect is a method for generating a learning model according to any one of the first aspect to the fifth aspect, and the bone data is data representing the skeleton of a golfer during a golf swing.

The learning model generating device according to the seventh aspect is a learning model generating device for analyzing the motion of an object, and includes a first acquisition unit that acquires bone data representing the skeleton of the object, a CG creation unit that creates three-dimensional computer graphics (CG) data representing the body shape of the object based on the bone data, an image creation unit that creates multiple depth images of the object based on the three-dimensional CG data, a second acquisition unit that acquires multiple motion values that represent the motion of one or more predetermined parts of the object, each corresponding to the multiple depth images, and a learning unit that trains a neural network that uses the depth images based on the three-dimensional CG data and multiple data sets of the corresponding motion values as training data, receives depth images of the object captured by a range image sensor as input, and outputs the motion values.

The motion analysis device according to the eighth aspect is a motion analysis device that analyzes the motion of an object, and includes an acquisition unit that acquires a depth image captured by a distance image sensor of the motion of the object, and a derivation unit that derives the motion value by inputting the acquired depth image to the neural network generated using a generation method according to any one of the first to seventh aspects.

A motion analysis method according to a ninth aspect is a motion analysis method for analyzing the motion of an object, and includes the following: A motion analysis program according to a tenth aspect causes a computer to execute the following:
(2-1) acquiring a depth image by capturing the motion of the object using a distance image sensor; (2-2) inputting the acquired depth image into the neural network generated using the generation method according to any one of the first to seventh aspects, thereby deriving the motion value.

According to the first aspect, three-dimensional CG data representing the body shape of an object is created based on bone data representing the skeleton of the object, and multiple depth images of the object are created based on such three-dimensional CG data. Then, a neural network is trained based on training data including such multiple depth images. As a result, it becomes possible to easily generate a highly accurate learning model.

1 is a diagram showing the overall configuration of a motion analysis system including a learning device for a learning model and a motion analysis device (computer) according to an embodiment of the present invention. FIG. 1 is a functional block diagram of a motion analysis system. A diagram showing the locations of markers attached to a golfer's body. FIG. 1 shows the model configuration of a neural network. 11 is a flowchart showing the flow of a motion analysis process based on a neural network. FIG. 1 illustrates an example of a normalized depth image. A graph of hip rotation angle estimates based on neural networks. 1 is a flowchart showing the flow of a learning process of a neural network. FIG. 4 is a diagram showing an example of bone data. FIG. 1 is a diagram showing an example of three-dimensional CG data. 13A and 13B are diagrams for explaining a method for deriving a waist rotation angle.

Below, a method and device for generating a learning model, as well as a motion analysis device, method, and program according to one embodiment of the present invention will be described with reference to the drawings. The following embodiment will be described using an example of a situation in which a golf swing is analyzed.

<1. Overview of the motion analysis system>
1 and 2 show an overall configuration diagram of a motion analysis system 100 including a computer 1 which is a motion analysis device according to this embodiment. The motion analysis system 100 is a system for capturing a swing motion of a golf club 5 by a golfer 7 as a depth image and analyzing the swing motion based on the depth image. The above capturing is performed by a distance image sensor 2. The computer 1 constitutes the motion analysis system 100 together with the distance image sensor 2, and analyzes the swing motion by analyzing the depth image acquired by the distance image sensor 2. The results of the analysis by the computer 1 are used for various purposes such as fitting a golf club 5 suitable for the golfer 7, improving the form of the golfer 7, and developing golf equipment.

The analysis of the swing motion is performed based on a neural network 8 that receives the depth image as an input. The neural network 8 is a learning model for analyzing the swing motion, and outputs a motion value _W1 that represents the swing motion. The neural network 8 is constructed by prior learning.

In the following, the motion analysis system 100 not only analyzes the motion based on the neural network 8, but also learns the neural network 8 (generates a learning model). That is, the computer 1 functioning as the motion analysis device according to this embodiment also functions as the learning model generation device according to this embodiment. However, such motion analysis processing and advance learning processing may be executed by separate hardware systems. In Figures 1 and 2, the hardware configuration used only in the learning processing, specifically, the motion capture system 3 and the angular velocity sensor 4, are shown by dotted lines (the motion capture system 3 is omitted in Figure 1). Therefore, after the neural network 8 is learned and stored in the memory unit 13, these measuring devices 3 and 4 can be omitted from the motion analysis system 100.

Below, we will first describe the details of each part of the motion analysis system 100, and then explain the model configuration of the neural network 8, the motion analysis process based on the neural network 8, and the learning process of the neural network 8.

<2. Details of each part>
<2-1. Range image sensor>
The distance image sensor 2 is a camera with a distance measuring function, and measures the distance to a golfer 7 who is test hitting a golf club 5. Therefore, the distance image sensor 2 can output a depth image. A depth image is an image in which data on the depth of a subject in the optical axis direction of the camera (depth data) is assigned to pixels within an imaging range perpendicular to the optical axis of the camera.

Although not limited thereto, the distance image sensor 2 used in this embodiment employs a method such as a time-of-flight method using infrared rays or a dot pattern projection method. Therefore, as shown in FIG. 1, the distance image sensor 2 has an IR light emitter 21 that emits infrared rays forward, and an IR light receiver 22 that receives the infrared rays irradiated from the IR light emitter 21 and reflected back from the subject. The IR light receiver 22 is a camera having an optical system and an image sensor. In the dot pattern projection method, the IR light receiver 22 reads the dot pattern of the infrared rays irradiated from the IR light emitter 21, detects the dot pattern by image processing inside the distance image sensor 2, and calculates the depth based on this. In this embodiment, the IR light emitter 21 and the IR light receiver 22 are housed in the same housing 20 and are arranged in front of the housing 20. In this embodiment, the distance image sensor 2 is installed in front of the golfer 7 to photograph the golfer 7 from the front side, and the IR light emitter 21 and the IR light receiver 22 are directed toward the golfer 7.

The distance image sensor 2 includes a built-in CPU 23 that controls the overall operation of the distance image sensor 2, as well as a built-in memory 24 that at least temporarily stores captured image data. A control program that controls the operation of the distance image sensor 2 is stored in the memory 24. The distance image sensor 2 also includes a built-in communication unit 25 that outputs captured image data to an external device such as the computer 1 via a wired or wireless communication line 17. In this embodiment, the CPU 23 and memory 24 are also housed in the housing 20, together with the IR light emitter 21 and the IR light receiver 22.

<2-2. Computer>
The configuration of the computer 1 will be described with reference to Fig. 2. The computer 1 is a general-purpose computer in terms of hardware, and is realized as, for example, a desktop computer, a laptop computer, a tablet computer, or a smartphone. The computer 1 is manufactured by installing a program 6 into the general-purpose computer from a computer-readable recording medium such as a CD-ROM or a USB memory, or via a network such as the Internet. The program 6 is software for analyzing a golf swing based on image data sent from the distance image sensor 2, and causes the computer 1 to execute operations described later. The program 6 also includes a program module for causing the computer 1 to execute a learning process described later.

The computer 1 comprises a display unit 11, an input unit 12, a memory unit 13, a control unit 14, and a communication unit 15. These units 11 to 15 are connected to each other via a bus line 16 and can communicate with each other. The display unit 11 can be configured as a liquid crystal display or the like, and displays the results of the golf swing analysis to the user. Note that the user here is a general term for anyone who requires the results of the golf swing analysis, such as the golfer 7 himself/herself, his/her instructor, golf equipment developers, etc. The input unit 12 can be configured as a mouse, keyboard, touch panel, etc., and accepts operations from the user on the computer 1.

The storage unit 13 may be configured with a hard disk or the like. The storage unit 13 stores a program 6. The storage unit 13 also stores information defining a neural network 8 that is learned in a learning process described later and is used in a motion analysis process described later. The control unit 14 may be configured with a CPU, a ROM, a RAM, or the like. The control unit 14 reads out and executes the program 6 in the storage unit 13, thereby virtually operating as a first acquisition unit 14a, a second acquisition unit 14b, a third acquisition unit 14c, a fourth acquisition unit 14d, a CG creation unit 14e, an image creation unit 14f, a learning unit 14g, a fifth acquisition unit 14h, a derivation unit 14i, and a display control unit 14j. Details of the operation of each unit 14a to 14j will be described later. The communication unit 15 functions as a communication interface that receives data from external devices such as the distance image sensor 2, the motion capture system 3, and the angular velocity sensor 4 via a communication line 17.

<2-3. Motion capture system>
In this embodiment, a motion capture system 3 is used to create teacher data used in the learning process. The motion capture system 3 is used only in the learning process. The motion capture system 3 includes a plurality of cameras 31, 31, .... The plurality of cameras 31, 31, ... are arranged at positions where the manner in which the golfer 7 swings the golf club 5 can be photographed from various directions so that the swing motion of the golfer 7 can be measured in three dimensions. As the motion capture system 3, for example, a three-dimensional motion analysis system manufactured by VICON Corporation can be preferably used.

In this embodiment, markers are attached to multiple predetermined parts of the body so that the cameras 31, 31, ... can easily capture the physical behavior of the golfer 7. More specifically, as shown in FIG. 3, light-reflective spherical markers 30, 30, ... are attached to predetermined parts of the golfer 7, such as the head, wrists, fingertips, elbows, shoulders, hips, knees, ankles, and toes. In this example, the markers 30, 30, ... are attached to a body suit worn by the golfer 7.

The cameras 31, 31, ... continuously capture images of the golfer 7 using the golf club 5 while he or she is performing a swing motion at a predetermined sampling frequency. The sampling frequency can be, for example, 500 Hz. In this embodiment, time-series image data is captured from at least the address, top, impact, and finish. The cameras 31, 31, ... are synchronized, and the golf swing viewed from various directions is captured at the same time.

Although not specifically shown, the motion capture system 3 is equipped with cameras 31, 31, ... as well as a computer that processes image data captured by the cameras 31, 31, ... and calculates position information (three-dimensional coordinates) of multiple markers captured in the image data. The motion capture system 3 is also equipped with a communication device for transmitting the calculated position information to the computer 1, which is an external device, via a communication line 17. The communication device can be wireless so as not to interfere with the swing motion, or can be connected to the computer 1 in a wired manner via a cable. The position information measured by the motion capture system 3 is transmitted to the computer 1 in real time via the communication device.

2-4. Angular velocity sensor
In this embodiment, in addition to the motion capture system 3, an angular velocity sensor 4 is also used to create teacher data used in the learning process. The angular velocity sensor 4 is also used only in the learning process. Although details will be described later, in this embodiment, the hip rotation motion of the golfer 7 during a swing motion is analyzed in the motion analysis process. For this reason, the angular velocity sensor 4 is attached to the hip of the golfer 7, and measures angular velocity data as a teacher signal representing the true value of the hip rotation angle. The angular velocity sensor 4 is a triaxial angular velocity sensor, and the angular velocity data here includes the value of the angular velocity in a three-dimensional orthogonal coordinate system.

The angular velocity sensor 4 continuously measures the state of the golfer 7 swinging the golf club 5 at a predetermined sampling frequency. The sampling frequency can be, for example, 1000 Hz. In this embodiment, the angular velocity sensor 4 measures time-series angular velocity data at least from the address to the finish.

Although not specifically shown, the angular velocity sensor 4 is also equipped with a communication device for transmitting angular velocity data to the computer 1, which is an external device, via a communication line 17. The communication device can be wireless so as not to interfere with the swing motion, or can be connected to the computer 1 in a wired manner via a cable. The time-series angular velocity data measured by the angular velocity sensor 4 is output to the computer 1 in real time via the communication device.

<3. Neural network model configuration>
4 shows a model configuration of the neural network 8. As described above, the neural network 8 is a network that receives a depth image as input and outputs a motion value _W1 representing a swing motion. In this embodiment, the neural network 8 analyzes the hip rotation motion of the golfer 7 during a swing motion, and more specifically, quantitatively derives the hip rotation angle as the motion value _W1 .

The model configuration of the neural network 8 can be designed in various ways and is not particularly limited, but in this embodiment, as shown in FIG. 4, it is designed as follows. The neural network 8 is a neural network called ResNet50, and has a layer configuration of 50 layers (excluding the batch normalization layer) in total, which includes a convolution layer (represented as COV in FIG. 4), a batch normalization layer (represented as batch NORM in FIG. 4), and a pooling layer, and then repeats a set of three convolution layers 16 times. In addition, in FIG. 4, the "number x number" attached to each convolution layer means the size of the kernel. In addition, since the data learned by ResNet50 becomes a residual, which is the difference between the input and the output, at the end of each set including three convolution layers, the input (represented by x: skip connection in FIG. 4) and the residual (represented by F(x) in FIG. 4) are added together. The final layer of ResNet50, a fully connected layer consisting of 2048 units, is connected to one output layer, and finally, a one-dimensional motion value _W1 is output.

4. Motion Analysis Processing Based on Neural Networks
The golf swing motion analysis process will be described below with reference to Fig. 5. As already mentioned, in this embodiment, a motion value _W1 that quantitatively represents the rotation angle of the hips of the golfer 7 during the swing motion is derived based on the neural network 8. The depth images to be analyzed are acquired in time series as moving images. Therefore, hereinafter, the depth images may be referred to as depth frames or simply as frames.

First, in step S1, the golfer 7 tries out the golf club 5, and the swing motion at this time is captured as a video by the distance image sensor 2. The time-series depth frames thus acquired are transmitted from the distance image sensor 2 to the computer 1. On the computer 1 side, the fifth acquisition unit 14h acquires these depth frames and stores them in the memory unit 13.

Then, the derivation unit 14i normalizes the depth frame at each timing during the swing motion (step S2). The normalization here refers to a process of scaling the gradation of the depth frame in accordance with the depth of the subject including the golfer 7. Specifically, the derivation unit 14i reads out the time-series depth frames stored in the storage unit 13. At this time, the depth data, which is the pixel value of the depth frame, has a gradation according to the standard of the distance image sensor 2, and in this embodiment, 16 bits are assigned to one pixel, and each pixel has a pixel value of 0 to 65535. The imaging range in the depth direction of the distance image sensor 2 is also determined by the standard of the distance image sensor 2. On the other hand, in order to estimate the rotation angle of the hips of the golfer 7, depth data other than that of the golfer 7 is not particularly required. Therefore, the gradation of the depth frame is scaled so that the pixel value included in the area capturing the golfer 7 (hereinafter referred to as the person area) in the depth frame has a value within the range of 0 to 65535.

The installation position of the distance image sensor 2 and the standing position of the golfer 7 are approximately fixed. Therefore, the range of values that the depth data can take in the person area (hereinafter referred to as the person depth range) can be set in advance. In this embodiment, the derivation unit 14i scales and converts the pixel value (depth data) of each pixel in the depth frame according to the following formula, where the person depth range is min _z to max _z , and z on the right side is the pixel value of each pixel in the depth frame. z on the left side is the pixel value after conversion.

The above scale conversion is a process for extracting an area having pixel values within the person depth range from the depth frame. Figure 6 shows a depth frame at a specific timing after the above scale conversion. As can be seen from the figure, in the depth frame after the scale conversion, a pixel value of "0" (black) is given to areas that mainly capture things other than the golfer 7, i.e., the background area. As a result, the above scale conversion extracts a person area.

Next, the derivation unit 14i extracts an area near the waist of the golfer 7 at each timing during the swing motion (hereinafter, waist area) from the depth frame normalized in step S2 (step S3). Specifically, the derivation unit 14i extracts a rectangular area circumscribing the person area from the depth frame of a predetermined size (512 x 424 pixels in this embodiment) normalized in step S2, and cuts out an area of a predetermined size (128 x 64 pixels in this embodiment) at a predetermined position within the rectangular area as the waist area. When a human body is captured from a predetermined angle during a swing motion, the range that the waist occupies within the entire human body is roughly determined, so the area within this range is cut out as the waist area.

The movement of the waist, which is of interest, is independent of the movement of other parts of the body, such as the shoulders, legs, and arms. Therefore, if the neural network 8 is trained based on an image showing the entire human body, there is a risk that the accuracy of detecting features corresponding to the appearance of the waist will decrease. Furthermore, if the image size is too large, analysis by the neural network 8 may become difficult. In step S3, the waist region is extracted from the depth frame to be analyzed so that the characteristics of waist movement can be detected accurately based on the neural network 8.

In the next step S4, the derivation unit 14i sequentially inputs the images of the waist region in time series during the swing motion extracted in step S3 to the neural network 8. As a result, the output layer of the neural network 8 sequentially outputs a motion value _W1 quantitatively expressing the swing motion, in this embodiment, the rotation angle _W1 of the waist of the golfer 7. Note that in this embodiment, the images of a predetermined size (128×64 pixels in this embodiment) acquired up to step S3 are compressed to a predetermined size (64×32 pixels in this embodiment) and then input to the neural network 8.

Next, the derivation unit 14i smoothes and interpolates the time series data of the rotation angle _W1 derived in step S4 (step S5). Fig. 7 shows an example of smoothing using a five-point moving average and performing spline interpolation to convert data at 33 ms intervals into data at 1 ms intervals. As a result, smooth, interpolated time series data of the rotation angle _W1 is obtained.

Thereafter, the display control unit 14j displays the rotation angle _W1 and its time series change before and after the smoothing and complementation of the time series during the swing motion derived in steps S4 and S5, as well as a graph of these as shown in Fig. 7 on the display unit 11 (step S6). This allows the user to understand the hip rotation motion of the golfer 7.

5. Neural network learning process
Next, the learning process of the neural network 8 will be described with reference to FIG. 8. In this learning process, teacher data is created and learning is performed based on the teacher data. In this embodiment, for transfer learning, two types of learning data sets are created as teacher data, and learning is performed based on each of them. Hereinafter, for the sake of distinction, the learning data set acquired in steps S11 to S14 may be called the first learning data set, and the learning data set acquired in steps S16 and S17 may be called the second learning data set. That is, in this embodiment, transfer learning is performed, and first, the parameters of the neural network 8 are learned based on the first learning data set (step S15), and then, further learning is performed based on the second learning data set (step S18) using the learned parameters as initial values.

Specifically, first, golfer 7 is asked to try out golf club 5. In step S11, the swing motion at this time is measured by motion capture system 3, and time-series position information of multiple markers 30 attached to golfer 7's body is obtained. Here, multiple cameras 31, 31, ... included in motion capture system 3 simultaneously capture images from different viewpoints, and three-dimensional coordinates are obtained as position information. The time-series position information of multiple markers 30 measured as described above is transmitted from motion capture system 3 to computer 1. On the computer 1 side, first acquisition unit 14a acquires this position information and stores it in memory unit 13.

The first acquisition unit 14a acquires time-series bone data D1 representing the body skeleton of the golfer 7 during a golf swing by sequentially processing the time-series position information of the multiple markers 30 stored in the memory unit 13. Bone data D1 is typically data that combines multiple joints and the bones that connect them, as shown in FIG. 9, and represents the skeleton of an object as a whole, and is sometimes called skeleton data. The first acquisition unit 14a creates bone data D1 by appropriately linking the three-dimensional coordinates of multiple specific parts of the golfer 7's body to which markers 30 are attached.

In the next step S12, the CG creation unit 14e sequentially optimizes the time-series bone data D1 to create time-series three-dimensional computer graphics (CG) data D2 (see FIG. 10) representing the body type of the golfer 7 during a golf swing. More specifically, the CG creation unit 14e applies body type data that defines a body type prepared in advance to the bone data D1, and reproduces the body type of the golfer 7 by fleshing out the skeleton represented by the bone data D1. Here, the same skeleton is fleshed out in different ways, and three-dimensional CG data D2 representing different body types is created from the bone data D1 representing the same skeleton. For example, three-dimensional CG data D2 corresponding to multiple body types, such as "fat," "normal," and "thin" types, is created from the bone data D1 representing the same skeleton.

In the next step S13, the image creation unit 14f creates multiple depth images of the body of the golfer 7 during a golf swing based on the time-series three-dimensional CG data D2. More specifically, depth images are created when the body of the golfer 7 represented by the three-dimensional CG data D2 is viewed from a predetermined direction. The predetermined direction here typically refers to the direction in which the distance image sensor 2 captures the swing motion of the golfer 7 in the motion analysis process.

In this embodiment, multiple depth images with different viewpoints are created from the three-dimensional CG data D2 of the same scene (timing). For example, multiple depth images corresponding to multiple viewpoints with the same direction relative to the golfer 7 but different distances to the golfer 7 are created from the three-dimensional CG data D2 representing the same scene. Also, multiple depth images corresponding to multiple viewpoints slightly shifted to the left or right from the above-mentioned specified direction are created from the three-dimensional CG data D2 representing the same scene (timing).

In this embodiment, each depth image created as described above is normalized in the same manner as steps S2 and S3 included in the motion analysis process, and an image of the waist region is extracted. That is, a depth image similar to the depth image input to the neural network 8 in the motion analysis process is prepared. Hereinafter, the image of the waist region normalized and obtained here may also be simply referred to as a depth image.

In the next step S14, the second acquisition unit 14b acquires a number of motion values _W1 corresponding to the number of depth images (normalized images of the waist region) created in step S13. The motion value _W1 here is the rotation angle of the waist of the golfer 7 during the golf swing, as described above. In this embodiment, the motion value _W1 here is derived based on the bone data D1 and/or the three-dimensional CG data D2. More specifically, the three-dimensional coordinates of the left hip joint and the right hip joint (shown by solid circles in Figs. 9 and 10) are acquired from the bone data D1 and/or the three-dimensional CG data D2, and the inclination θ of the straight line connecting these in a plan view with respect to a reference line is derived as shown in Fig. 11, and this is set as the rotation angle _W1 of the waist. The second acquisition unit 14b sequentially derives motion values W1 from the time-series bone data D1 acquired in step S11 and/or the time-series three-dimensional CG data D2 created in step S12, and associates each motion value _W1 with a depth image created from the same data D1 and/or D2 in step S13. This allows a large number of first learning data sets to be prepared, each of which is _a combination of a depth image based on the three-dimensional CG data D2 and a corresponding motion value _W1 .

In the next step S15, the learning unit 14g trains the neural network 8 using a large number of first learning data sets as teacher data. More specifically, the learning unit 14g inputs a depth image (a normalized image of the waist region) included in the first learning data set to the current neural network 8, obtains a motion value _W1 as an output value, and updates the parameters of the neural network 8 so as to minimize the error between this motion value _W1 and the motion value _W1 included in the same first learning data set. Then, the neural network 8 is optimized while successively applying the large number of first learning data sets created in steps S11 to S14.

In the next step S16, the third acquisition unit 14c acquires a large number of depth images captured by the distance image sensor 2 during the golf swing by the golfer 7. Specifically, similar to step S11, the golfer 7 is made to test hit the golf club 5. In step S16, the swing motion at this time is captured as a video by the distance image sensor 2. The time-series depth frames measured as described above are transmitted from the distance image sensor 2 to the computer 1. On the computer 1 side, the third acquisition unit 14c acquires this image data and stores it in the memory unit 13. Note that the measurement by the distance image sensor 2 in step S16 may be performed on the golf swing by the same golfer 7 at the same time as the measurement in step S11.

In this embodiment, each depth image acquired as described above is normalized in the same manner as steps S2 and S3 included in the motion analysis process, and an image of the waist region is extracted. That is, a depth image similar to the depth image input to the neural network 8 in the motion analysis process is prepared. Hereinafter, the image of the waist region normalized and acquired here may also be simply referred to as a depth image.

In the next step S17, the fourth acquisition unit 14d acquires a number of motion values _W1 corresponding to the number of depth images (normalized images of the waist region) acquired in step S16. The motion value _W1 here is the rotation angle of the waist of the golfer 7 during the golf swing, as described above. In this embodiment, the motion value _W1 here is acquired based on the time-series angular velocity data measured by the angular velocity sensor 4. More specifically, while the golf swing is being photographed by the distance image sensor 2 in step S16, the angular velocity sensor 4 also performs measurement. At this time, the angular velocity sensor 4 continuously measures angular velocity data representing the rotation angle of the waist of the golfer 7 during the golf swing. The time-series angular velocity data thus measured is transmitted from the angular velocity sensor 4 to the computer 1. On the computer 1 side, the fourth acquisition unit 14d acquires this angular velocity data and stores it in the memory unit 13.

The fourth acquisition unit 14d associates, with each depth image (normalized image of the waist region) acquired in step S16, angular velocity data acquired at the timing when the depth image was captured by the range image sensor 2. In this way, a large number of second learning data sets, which are combinations of depth images captured by the range image sensor 2 and the corresponding motion values _W1 , are prepared.

In the next step S18, similarly to step S15, the learning unit 14g trains the neural network 8 using a large number of second learning data sets as teacher data. More specifically, the learning unit 14g inputs a depth image (a normalized image of the waist region) included in the second learning data set to the current neural network 8, obtains a motion value _W1 as an output value, and updates the parameters of the neural network 8 so as to minimize the error between this motion value _W1 and the motion value _W1 included in the same second learning data set. Then, the neural network 8 is optimized while successively applying the large number of second learning data sets created in steps S16 and S17. This completes the learning process.

6. Modifications
Although several embodiments of the present invention have been described above, the present invention is not limited to the above-mentioned embodiments, and various modifications are possible without departing from the spirit of the present invention. For example, the following modifications are possible. In addition, the gist of the following modifications can be combined as appropriate.

<6-1>
In the above embodiment, a golf swing is analyzed, but the subject of analysis is not limited to this, and the swing motion of a hitting tool in tennis, baseball, etc. can also be analyzed in a similar manner. Also, instead of or in addition to the motion of the human body included in the swing motion, the motion of a hitting tool included in the swing motion can be the subject of analysis. Also, the subject of analysis does not have to be the motion of a sport or the motion of the human body, and any motion of any object can be the subject of analysis.

<6-2>
The motion value _W1 output from the neural network 8 is not limited to the rotation angle of the part of interest, and may be the position (coordinates), rotation speed, rotation acceleration, speed and acceleration, etc. of the part of interest. Furthermore, the part of interest is not limited to the waist of the golfer 7, and may be the shoulders, arms, head, center of gravity, etc. of the golfer 7. Furthermore, motion values _W1 of a plurality of parts may be derived as the output of the neural network 8. For example, a set of position information (coordinates) of a number of predetermined parts of the body may be derived, or bone data including position information (coordinates) of a number of joints in the body may be derived.

For example, consider the case where the shoulder rotation angle is derived as the motion value _W1 . In this case, in the motion analysis process, an image of the area near the shoulder (shoulder area) is extracted from the depth frame in step S3, and this is input to the neural network 8 in step S4. In the learning process preceding this, a depth image of the shoulder area is prepared in step S13, and the shoulder rotation angle is derived from the bone data D1 and/or the three-dimensional CG data D2 as the corresponding motion value _W1 in step S14. At this time, the three-dimensional coordinates of the left shoulder and the right shoulder are obtained from the bone data D1 and/or the three-dimensional CG data D2, and the inclination θ of the straight line connecting these in a planar view with respect to the reference line is derived, and this can be set as the shoulder rotation angle _W1 . Furthermore, in step S16, a depth image of the shoulder area is also prepared, and the shoulder rotation angle is obtained as the corresponding motion value _W1 in step S17. The shoulder rotation angle here can be obtained from the angular velocity sensor 4, for example, by attaching the angular velocity sensor 4 from the waist of the golfer 7 to the shoulder.

As in the above example, when position information (two-dimensional or three-dimensional coordinates) of a number of joints in the body is derived as the motion value W ₁ in the motion analysis process, these position information can be used to derive position information (two-dimensional or three-dimensional coordinates) of the center of gravity of the body. In this case, in the motion analysis process, in step S3, an image of the area (person area) mainly occupied by the golfer is extracted from the depth frame, and in step S4, this is input to the neural network 8. In addition, in the learning process preceding this, in step S13, a depth image of the person area is prepared, and in step S14, position information of a number of joints in the body corresponding to this is derived from the bone data D1 and/or the three-dimensional CG data D2 as the motion value W ₁ corresponding to this. Furthermore, in step S16, a depth image of the person area is also _prepared , and in step S17, position information of a number of joints in the body corresponding to this may be obtained based on the result of measurement by the motion capture system 3 performed simultaneously with the measurement by the distance image sensor 2, for example.

Then, based on the position information of many joints derived as the motion value W ₁ in the motion analysis process, the position information of the center of gravity of the body is derived. At this time, for example, the body is virtually divided into 14 parts (hereinafter referred to as divided parts) including the head, the torso, the left and right upper arms, the left and right forearms, the left and right hands, the left and right thighs, the left and right lower legs, and the left and right feet. Then, the mass ratio of each part to the whole body and the center of gravity position ratio, which is the ratio that defines the position where the center of gravity exists in each part, are determined in advance for each gender. Then, the position information (coordinates) of the center of gravity of the body can be derived from these mass ratios and center of gravity position ratios and the motion value W ₁ derived in the motion analysis process. More specifically, for example, the center of gravity position ratio of the head is 46%. In this case, the x coordinate of the center of gravity of the head is derived as (x coordinate of the apex of the head) + 0.46 × (x coordinate of the apex of the head - x coordinate of the neck), and the y and z coordinates of the center of gravity of the head are derived in the same manner. The x, y, and z coordinates of the vertex of the head and the neck are included in the motion value _W1 . Similarly, the three-dimensional coordinates of the center of gravity of the 14 divided parts are calculated. Here, the mass ratios of the 14 divided parts are respectively _m1 , _m2 , ..., _m14 . Also, the three-dimensional coordinates of the center of gravity of the 14 divided parts are respectively ( _x1 , _y1 , _z1 ), ( _x2 , _y2 , _z2 ), ..., ( _x14 , _y14 , _z14 ). At this time, the three-dimensional coordinates of the center of gravity of the body are calculated as follows.

<6-3>
In the above embodiment, the bone data is obtained from the position information obtained by the motion capture system 3, but for example, Kinect (registered trademark), which is one of distance image sensors, has a function of deriving skeleton data from a depth frame and outputting it together with the depth frame. Therefore, such skeleton data can also be used as the above bone data.

<6-4>
In the above embodiment, the image input to the neural network 8 is one depth image. However, a plurality of depth images in a time series may be input to the neural network 8 as a set. For example, three depth images captured consecutively can be combined into one set, such as RGB. In this case, the neural network 8 can learn by taking into account the time change of the motion value _W1 , and the prediction accuracy of the motion value _W1 is improved.

1. Computer (learning model generation device, motion analysis device)
14a: First acquisition unit 14b: Second acquisition unit 14c: Third acquisition unit 14d: Fourth acquisition unit 14e: CG creation unit 14f: Image creation unit 14g: Learning unit 14h: Fifth acquisition unit (acquisition unit)
14i Derivation unit 14j Display control unit 2 Distance image sensor 3 Motion capture system 4 Angular velocity sensor 5 Golf club 6 Program (motion analysis program)
7. Golfer (object)
8 Neural network 100 Motion analysis system D1 Bone data D2 Three-dimensional CG data

Claims (9)

A method for generating a learning model for analyzing an object's motion, comprising the steps of:
obtaining bone data representative of a skeleton of the object;
creating three-dimensional computer graphics (CG) data representing a body shape of the object based on the bone data;
generating multiple depth images of the object based on the 3D CG data;
obtaining a number of motion values representing motion of one or more predetermined portions of the object, the motion values corresponding to the number of depth images respectively;
and training a neural network that uses a large number of data sets of the depth images based on the three-dimensional CG data and the corresponding action values as training data, inputs a depth image of the object captured by a range image sensor, and outputs the action values ;
Creating the multiple depth images includes creating multiple depth images having different viewpoints based on the 3D CG data of a single scene.
How to generate a learning model. obtaining the multiple motion values includes deriving the multiple motion values based on at least one of the bone data and the three-dimensional CG data;
A method for generating a learning model according to claim 1 . Creating the three-dimensional CG data includes creating the three-dimensional CG data representing different body types by fleshing out the same skeleton in different ways.
A method for generating a learning model according to claim 1 or 2. acquiring multiple depth images of the object captured by the range image sensor;
acquiring a number of the motion values corresponding to the number of depth images captured by the distance image sensor;
The method further includes: training the neural network using the depth image based on the three-dimensional CG data and the multiple data sets of the corresponding action values as training data, and then training the neural network using the depth image captured by the distance image sensor and the multiple data sets of the corresponding action values as training data.
A method for generating a learning model according to any one of claims 1 to 3 . The bone data is data representing a golfer's skeleton during a golf swing.
A method for generating a learning model according to any one of claims 1 to 4 . A device for generating a learning model for analyzing the motion of an object, comprising:
a first acquisition unit that acquires bone data representing a skeleton of the object;
a CG creation unit that creates three-dimensional computer graphics (CG) data representing a body shape of the object based on the bone data;
an image creation unit that creates multiple depth images of the object based on the three-dimensional CG data;
A second acquisition unit that acquires a number of motion values that represent motions of one or more predetermined parts of the object, the motion values corresponding to the number of depth images, respectively;
a learning unit that uses a large number of data sets of the depth images based on the three-dimensional CG data and the corresponding action values as training data, inputs a depth image of the object captured by a distance image sensor, and trains a neural network that outputs the action values;
the image creation unit creates a plurality of depth images having different viewpoints based on the 3D CG data of the same scene;
A device for generating learning models. A motion analysis device for analyzing a motion of an object, comprising:
an acquisition unit that acquires a depth image obtained by capturing the motion of the object using a distance image sensor;
and a derivation unit that derives the operation value by inputting the acquired depth image to the neural network generated by using the generation method according to any one of claims 1 to 5 .
Motion analysis device. A motion analysis method for analyzing a motion of an object, comprising:
acquiring a depth image by capturing the motion of the object using a distance image sensor;
and deriving the motion value by inputting the acquired depth image into the neural network generated using the generation method according to any one of claims 1 to 5 .
Motion analysis method. A motion analysis program for analyzing a motion of an object,
acquiring a depth image by capturing the motion of the object using a distance image sensor;
and deriving the motion value by inputting the acquired depth image into the neural network generated using the generation method according to any one of claims 1 to 5 .
Motion analysis program.