CN116343332A - An intelligent table tennis coaching method and system thereof - Google Patents

Abstract

The present invention relates to a kind of intelligent table tennis coaching method and system thereof, described method comprises: (1) carry out real-time data collection through embedded equipment, obtain motion data; (2) obtain the continuous skeleton image that contains time series information of student; (3) ) Compare the image with the action embedding vector in the model library to identify the action category; (4) Bind the motion data acquired by the embedded device with the identified action category; (5) Analyze and compare the final data. The system includes an embedded device, a binocular camera, and a terminal; the embedded device, the binocular camera, and the terminal are connected through a wireless communication module for data transmission. The present invention acquires the student's movements and the whole training data, and through intelligent identification and comparison, gives action standard judgment results and guidance suggestions based on the comparison results; the present invention is used for correcting and improving the basic movements of table tennis, and can replace the coach for the students Action teaching, guidance and error correction.

Description

An intelligent table tennis coaching method and system thereof

technical field

The invention relates to the field of artificial intelligence, in particular to an intelligent table tennis coaching method and system thereof.

Background technique

Learning how to play table tennis is a very difficult task for a beginner. In the existing table tennis teaching process, more video teaching, auxiliary tools or coaches are used to teach in person, such as using various auxiliary tools, such as walls, self-training equipment for self-training, such as hiring coaches to teach them table tennis Knowledge and technology, such as using teaching videos to learn swing movements, because the training process cannot be digitized, so the intelligent system has no effective application in it for the time being.

In the existing table tennis teaching, there is no perfect technology and product that can use modern scientific and technological means to collect the students' table tennis training situation in real time and accurately, and at the same time scientifically and effectively guide the basic learning and training of table tennis students. Especially for the basic learning stage of table tennis, there is an urgent need for a real-time, accurate and comprehensive table tennis teaching system, which can assist or even replace coaches in guiding students and correcting errors.

Contents of the invention

The embodiment of the present invention provides an intelligent table tennis coaching method and system thereof. In order to provide a basic understanding of some aspects of the disclosed embodiments, a brief summary is presented below. This summary is not an overview, nor is it intended to identify key/critical elements or delineate the scope of these embodiments. Its sole purpose is to present some concepts in a simplified form as a prelude to the more detailed description that is presented later.

The embodiment of the present invention provides a kind of intelligent table tennis coaching method, and its improvement is, comprises:

(1) Real-time data collection through embedded devices to obtain motion data;

(2) Obtain continuous skeleton images containing time series information of students;

(3) Compare the image with the action embedding vector in the model library to identify the action category;

(4) Bind the motion data acquired by the embedded device with the recognized action category;

(5) Analyze and compare the final data.

Preferably, said step (1) comprises that the sensor passes through the acceleration data of the table tennis racket, the gyroscope data and and based on the bottom data of the table tennis racket to establish a Cartesian coordinate system, and obtains the coordinate data of the table tennis racket through the coordinate data of three X, Y and Z axes. The tilt angle, and the data obtained by the pressure sensor when hitting the ball; wherein, the embedded device includes a SensorTile sensor for collecting accelerometer data, gyroscope data and pressure sensor data.

Preferably, said step (2) includes establishing a pre-training model for character detection, detecting each key action of the student in hitting the ball, detecting and framing the student in real time, generating continuous skeleton images containing time series information, and using the collected data as a training set , frame the student and the table tennis bat, detect the batter's batting action of the student holding the table tennis bat, and detect the key points of the batter.

Further, it includes obtaining motion images with de-artifacts after preprocessing, normalizing and normalizing them to [0, 1], and grouping the obtained images; 70% of them are used as training sets, and 20% of them are used as training sets. % is the test set and 10% is the validation set.

Further, the key points include left and right shoulders, left and right elbows, left and right wrists, left and right index fingers, and thumb joints.

Preferably, the model library includes, according to professional table tennis coaches, demonstrations of professional table tennis catching movements, including looping, attacking, pushing, rubbing, picking, twisting and other techniques, and through key Point detection corresponds to the key point coordinates of the action, and at the same time calculates the standard model library established by the Euclidean distance between the coordinates of each joint point during the corresponding action.

Preferably, said step (3) includes inputting the continuous bone images containing timing information detected in real time into the action feature extraction model to generate corresponding action key point embedding vectors; By comparing the action embedding vectors, the Euclidean distance or cosine distance between the feature vectors is used to select the action category that is recognized in real time by the bottom library features corresponding to the embedding vectors whose Euclidean distance or cosine distance is within the preset range.

Preferably, the step (4) includes binding the corresponding data obtained by the sensor at this time when detecting each key action of the student in the step (2), and each action of hitting the ball includes the hitting action Whether the action is standardized or not, the acceleration when hitting the ball, the hitting force of the table tennis racket, and the tilt angle of the table tennis racket calculated through the three-dimensional coordinates.

Preferably, said step (5) includes acquiring the data transmission terminal of the embedded device, displaying the data of each batting action and corresponding batting through the terminal gui interface, and the user or the table tennis coach real-time or feedback through the gui interface See whether the batter's movement is standard when hitting the ball, as well as the strength of the ball, and display the standard movement, assisting teaching to improve the table tennis level of students.

The present invention also provides a kind of intelligent table tennis coaching system based on another purpose, and its improvement is that, described system comprises embedded device, binocular camera, and terminal;

The embedded device includes a SensorTile sensor for collecting accelerometer data, gyroscope data, and pressure sensor data;

The binocular camera is used to obtain the video of students playing table tennis in real time;

The terminal includes a processor for data processing and a gui interface for data display;

The embedded device, the binocular camera, and the terminal are connected through a wireless communication module for data transmission.

preferred,

The SensorTile sensor includes a plurality of MEMS sensor modules, and the SensorTile sensor is embedded in a table tennis handle powered by a micro battery, and data transmission uses a system-on-chip radio frequency transceiver circuit;

The wireless communication module adopts a USR-WIFI232-S wireless transceiver chip, which is connected to a processor through a UART, and the processor processes data.

The technical solutions provided by the embodiments of the present invention may include the following beneficial effects:

The present invention obtains the student's movements and the whole training data, and through intelligent identification and comparison, gives action standard judgment results and guidance suggestions based on the comparison results; the present invention is used for correcting and improving the basic movements of table tennis, and can replace the coach for the students Action teaching, guidance and error correction.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention.

Description of drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and together with the description serve to explain the principles of the invention.

Fig. 1 is a schematic flowchart of an intelligent table tennis coaching method according to an exemplary embodiment.

Fig. 2 is a schematic diagram of key points detected in the flow of an intelligent table tennis coaching method according to an exemplary embodiment.

Fig. 3 is a schematic diagram of establishing a Cartesian coordinate system in the flow of an intelligent table tennis coaching method according to an exemplary embodiment.

Fig. 4 is a schematic diagram of a gui interface in an intelligent table tennis coaching system according to an exemplary embodiment.

Detailed ways

The following description and drawings illustrate specific embodiments of the invention sufficiently to enable those skilled in the art to practice them. The examples merely represent possible variations. Individual components and functions are optional unless explicitly required, and the order of operations may vary. Portions and features of some embodiments may be included in or substituted for those of other embodiments. The scope of embodiments of the present invention includes the full scope of the claims, and all available equivalents of the claims. Herein, various embodiments may be referred to individually or collectively by the term "invention", which is for convenience only and is not intended to automatically limit the scope of this application if in fact more than one invention is disclosed. A single invention or inventive concept. Herein, relational terms such as first and second etc. are used only to distinguish one entity or operation from another without requiring or implying any actual relationship or relationship between these entities or operations. order. Furthermore, the term "comprises", "comprises" or any other variation thereof is intended to cover a non-exclusive inclusion such that a process, method or apparatus comprising a set of elements includes not only those elements but also other elements not expressly listed element. Various embodiments herein are described in a progressive manner, each embodiment focuses on the differences from other embodiments, and the same and similar parts of the various embodiments may be referred to each other. As for the structures, products, etc. disclosed in the embodiments, since they correspond to the parts disclosed in the embodiments, the description is relatively simple, and for relevant parts, please refer to the description of the method part.

The present invention will be further described below in conjunction with accompanying drawing and embodiment:

As shown in Figure 1, the invention provides a kind of intelligent table tennis coaching method, comprising:

(1) Real-time data collection through embedded devices to obtain motion data;

Since gesture recognition is difficult to obtain data such as the power of the batter and the angle coordinates of the racket's inclination, the corresponding data is recorded through the embedded device in the table tennis racket. The embedded device is the SensorTile sensor embedded in the table tennis racket, SensorTile Sensors can collect accelerometer data, gyroscope data, pressure sensor data, etc. The SensorTile sensor contains multiple MEMS sensor modules. When the physical sensor moves, the silicon block will also move to collect real-time data. The sensor is embedded in the table tennis handle and powered by a micro battery. The data transmission is designed using the system-on-chip (SOC) method. RF transceiver circuit. The wireless communication module adopts USR-WIFI232-S wireless transceiver chip. The maximum distance and rate of data transmission are both 50m, exceeding 1mb/s. The module is connected to the processor through UART, and the processor processes the data to obtain various motion parameters of the table tennis racket combined with attitude estimation, and then obtain real-time motion data of the students. After the data obtained by the sensor is processed, a power and an angle when the batter hits the ball can be obtained.

Initialize the sensor before hitting the ball so the MEMS sensor can start retrieving real-time data. The present invention uses a frequency of 10 Hz for data collection, so that the SensorTile sensor will not be overloaded. The MEMS sensor acquires data from the environment. The sensor establishes a Cartesian coordinate system based on the acceleration data of the table tennis racket, gyroscope data, and the bottom data of the table tennis racket, as shown in Figure 3. It is obtained by coordinate data of three X, Y, and Z axes. The tilt angle of the table tennis racket, and the data obtained by the pressure sensor when hitting the ball.

(2) Obtain continuous skeleton images containing time series information of students;

Establish a pre-training model for character detection, and obtain real-time video of students playing table tennis through binocular cameras. In order to estimate the pose more accurately and avoid other data noise interference, the present invention uses a target detection algorithm to detect students , the pre-trained model of person detection in the COCO data set loaded by the yoloV5 model can detect the ping-pong students in real time, and use the boundingbox to frame the students. Detect each key movement of a student hitting the ball, detect and frame the student in real time, generate a continuous skeleton image containing time series information, collect data as a training set, frame the student and the table tennis racket, and detect the student holding the table tennis racket to hit the ball The player's batting action is used to detect the key points of the player.

Among them, the situation of multiple people may be detected during detection. The present invention collects the picture data of 2000 students including holding table tennis bats as a training set, and the labeling personnel use lableme software to frame the students and table tennis bats, and send them into the yolov5 algorithm Training, the number of training is 2000 epoch, and then use the trained yolov5 to directly detect the students holding the table tennis racket, and at the same time, in order to be able to automatically detect the hitting action of the batter. The present invention is accomplished through 1. preprocessing: motion artifact detection and artifact removal, and 2. pri_trans_cpn network introducing attention mechanism and prior knowledge.

1. Pretreatment:

Since the input image is the bounding box containing the human body detected by yolov5, the input image is scaled to 520*520 by opencv's resize, and the input image is detected by using the Laplacian second-order differential linear operator (formula (1)) When the high-frequency component is less, it means that the image is relatively blurred and there are artifacts. Specifically, the variance of the image is calculated through the Laplacian filter, and the threshold is set to 0.3. When the variance is less than the threshold, it is determined that the The image is blurry with motion artifacts.

For images with motion artifacts, it is because the original image has moved a distance of L pixels in the horizontal or vertical direction. Here, it can be understood that the original image has made a Fourier blur kernel of size L in the horizontal or vertical direction. Transformation, assuming that the exposure time of the camera is t, the moving distance of the portrait is decomposed into x(t) and y(t) according to the horizontal and vertical. The original image is f(x, y), and the blurred image containing motion artifacts is g(x, y), which is obtained by Fourier transform of the original image, as shown in formula (2).

The formula is obtained by Fourier transforming g(x,y)

则G(u,v)＝J(u,v)k(u，v)，这里的G(u，v)为变换的频谱幅度，使用radon变换计算出频率偏移角度即模糊核的角度，并根据图像模糊后相邻波纹间距得到偏移尺度，最后经过傅里叶逆变换后即可得到无运动伪影的清晰图像。suppose

Then G(u,v)=J(u,v)k(u,v), where G(u,v) is the transformed spectrum amplitude, using radon transformation to calculate the frequency offset angle, that is, the angle of the blur kernel, And the offset scale is obtained according to the distance between adjacent corrugations after the image is blurred, and finally a clear image without motion artifacts can be obtained after inverse Fourier transform.

2. Introduce the pri_trans_cpn network of attention mechanism and prior knowledge;

This part of the network uses the cpn (cascaded pyramid network) network as the basic structure of the network. The traditional cpn network is mainly composed of GlobalNet and RefineNet. It is difficult to achieve better key point prediction in some fuzzy parts or occluded areas. The cpn network combined with the transformer network can better detect key points in occluded or blurred areas through the attention mechanism of the transformer network. At the same time, a large number of key points of the human body in easily occluded areas are added to the network layer as prior knowledge for training, thereby finally providing key point detection accuracy.

The key point detection algorithm detects the key points of the hitter detected in the bounding box, and uses the bounding box of the target detection as input, which belongs to a top-down key point detection algorithm. The algorithm framework process includes: using the target The detected RGB image with a size of w*h in the bounding box is used as input, the key points of the human body in the image are located, and the key points are linked through the human body structure to display the skeletal structure of each person. Due to the large changes in the movement of table tennis, there may be some key points of the human body that are blocked, resulting in the inability to accurately identify the movement. The target has motion artifacts, resulting in inaccurate detection.

The specific operation is:

1) Obtain the de-artifact moving image after preprocessing, and normalize it to [0, 1].

2) Data grouping of the obtained images, wherein 70% is used as a training set, 20% is a test set, and 10% is a verification set.

3) The preprocessed training set and labels are input to the downsampling network composed of 4 resnet units. The original image size is (x, y), and the size after four downsamplings is (x/4, y/4) , (x/8, y/8), (x/16, y/16), (x/32, y/32), and convert the channels output by each layer to the same number of channels, resulting in 4 Feature maps with inconsistent dimensions and the same number of channels, the prior network part uses the key point heat map with difficult detection parts as prior knowledge and scales to (x/4, y/4), (x/8, y/8) ), (x/16, y/16), (x/32, y/32) size, and the output of each layer of GlobalNet is added element-wise.

Use L2 loss, that is, the output of the network and the label to calculate the loss of all key points. In order to prevent the drift of some key points under high-speed movement, the transformer network is added to encode the position of key points. Transformer is a network based on encoding-decoding The structure, which contains multiple multi-head attention layers, is used to capture the relationship between different position codes of input information, which can make the model focus on different position information.

Divide the feature map output after adding prior knowledge into 9 patches in equal proportions, and add position codes from 1 to 9, flatten the multiple patches through the flatten layer and input them into the transformer network, and use L2 loss as the loss after the output is obtained The function, the formula is as follows, that is, calculate the output of the network and label to calculate the loss of all key points:

After GlobalNet adds prior knowledge and adds position encoding output through transformer, it connects different numbers of botleneck blocks to each layer of feature maps, and then undergoes upsampling at different magnifications, and uses bilinear interpolation for upsampling. After restoring the size of the feature map and then performing the concat operation, the combination of features of different scales is achieved. After concat output, calculate the loss of all key points with the label, then sort the loss from large to small, and finally select the top-k losses The backpropagation used for the network finally passes through a bottlenet block to reduce the number of channels. Here, 1*1 convolution is used, each of which undergoes a simple transformation, and finally connected to the softmax function to obtain the final output of the network.

(3) Compare the image with the action embedding vector in the model library to identify the action category;

Since the key points of hitting the ball mainly lie in the transformation of bone points such as wrists and arms, the key points selected for detection are points 11-24 in Figure 2, including left and right shoulders, left and right elbows, left and right wrists, left and right index fingers, and thumb joints.

According to professional table tennis coaches, demonstrate professional table tennis ball receiving actions, including loop ball, attack ball, push and push, rub ball, pick, twist and pull and other techniques, and detect the key point coordinates of corresponding actions through key points , and at the same time calculate the Euclidean distance between the coordinates of each joint point during the corresponding action, and save it as a standard model library.

The real-time detection of continuous bone images containing time series information is input into the action feature extraction model to generate corresponding action key point embedding vectors. By comparing the action key point embedding vector with the action embedding vector in the established standard model library, through the Euclidean distance or cosine distance between feature vectors, select the embedding vector corresponding to the Euclidean distance or cosine distance within the preset range. The action category recognized by the bottom library features in real time.

(4) Bind the motion data acquired by the embedded device with the recognized action category;

When detecting each key movement of the student in step (2), the corresponding data obtained by the sensor at this time is bound at the same time, that is, whether each movement contains the specification of the movement, and the hitting The acceleration during the action, the hitting force of the table tennis racket, and the inclination angle of the table tennis racket calculated by the three-dimensional coordinates.

(5) Analyze and compare the final data.

Write all the data obtained by the embedded device into the terminal through the wireless transmission module. The terminal is equipped with a gui interface, and the data of each hitting action and the corresponding hitting time are displayed in the gui interface. The user or coach can use the gui interface in real time or Look back at whether the batter’s movement is standard when hitting the ball, and the hitting strength is shown in Figure 4, and the standard movement is displayed, so as to assist teaching and quickly improve the table tennis level of students.

Among them, the student or coach mobile terminal can directly connect to the terminal and watch through the GUI interface of the mobile APP;

The present invention also provides an intelligent table tennis coaching system, comprising: an embedded device, a binocular camera, and a terminal;

The embedded device includes a SensorTile sensor for collecting accelerometer data, gyroscope data, and pressure sensor data;

The binocular camera is used to obtain the video of students playing table tennis in real time;

The terminal includes a processor for data processing and a gui interface for data display;

The embedded device, the binocular camera, and the terminal are connected through a wireless communication module for data transmission. The corresponding data is recorded by the embedded device in the table tennis racket. The embedded device is the SensorTile sensor embedded in the table tennis racket. The SensorTile sensor can collect accelerometer data, gyroscope data, pressure sensor data, etc. The SensorTile sensor contains multiple MEMS sensor modules. When the physical sensor moves, the silicon block will also move to collect real-time data. The sensor is embedded in the table tennis handle and powered by a micro battery. The data transmission is designed using the system-on-chip (SOC) method. RF transceiver circuit. The wireless communication module adopts USR-WIFI232-S wireless transceiver chip. The maximum distance and rate of data transmission are both 50m, exceeding 1mb/s. The module is connected to the processor through UART, and the processor processes the data to obtain various motion parameters of the table tennis racket combined with attitude estimation, and then obtain real-time motion data of the students. After the data obtained by the sensor is processed, a power and an angle when the batter hits the ball can be obtained.

Among them, the development of the present invention is based on Windows 11, python 3.9 is used for gesture recognition, and the graphics card is RTX3090.

It should be understood that the present invention is not limited to the processes and structures that have been described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from the scope thereof. The scope of the invention is limited only by the appended claims.

Claims (10)

1. An intelligent table tennis coaching method, comprising:

(1) Collecting real-time data through embedded equipment to obtain motion data;

(2) Acquiring continuous skeleton images of a student containing time sequence information;

(3) Comparing the image with the action embedded vector in the model library to identify action categories;

(4) Binding the motion data acquired by the embedded equipment with the identified action category;

(5) And (5) analyzing and comparing the final data.

2. The intelligent table tennis training method according to claim 1, wherein the step (1) comprises the steps that the sensor establishes a cartesian coordinate system through acceleration data, gyroscope data and based on bottom data of the table tennis bat, and obtains the inclination angle of the table tennis bat through three X, Y and Z-axis coordinate data, and data obtained by the pressure sensor during batting; the embedded device comprises a sensor and is used for collecting accelerometer data, gyroscope data and pressure sensor data.

3. The intelligent table tennis training method according to claim 1, wherein the step (2) comprises the steps of establishing a pre-training model of character detection, detecting key actions of each batting of a student, detecting and framing the student in real time, generating continuous skeleton images containing time sequence information, framing the student and a table tennis bat by collecting data as a training set, detecting batting actions of the student holding the table tennis bat to the batting person, and detecting key points of the batting person; the key points comprise left and right shoulders, left and right elbows, left and right wrists, left and right index finger joints and thumb joints.

4. An intelligent table tennis training method according to claim 3 comprising obtaining a motion image from which artifacts have been removed by preprocessing, normalizing and normalizing to between 0,1, and grouping the obtained images into data; wherein 70% is used as training set, 20% is used as test set, and 10% is used as verification set.

5. The intelligent table tennis training method according to claim 1, wherein the model library comprises a standard model library which is established according to professional table tennis training, demonstrates professional table tennis ball receiving actions including loop ball, attack ball, push, play, twist and other techniques, detects key point coordinates of corresponding actions through key points, and calculates euclidean distances between the coordinates of each joint point in the corresponding actions.

6. The intelligent table tennis training method according to claim 1, wherein the step (3) comprises inputting the continuous skeleton image containing the time sequence information detected in real time into an action feature extraction model to generate a corresponding action key point embedding vector; and comparing the action key point embedded vector with the action embedded vector in the established standard model library, and selecting the action category identified in real time by the base features corresponding to the embedded vector with the Euclidean distance or cosine distance within a preset range through the Euclidean distance or cosine distance between the feature vectors.

7. The intelligent table tennis training method according to claim 1, wherein the step (4) comprises binding the data obtained by the sensor when the key action of each ball is detected in the step (2), wherein each ball striking action comprises the specification of the ball striking action, the acceleration of the ball striking action, the ball striking force of the table tennis bat and the tilt angle of the table tennis bat calculated by three-dimensional coordinates.

8. The intelligent table tennis training method according to claim 1, wherein the step (5) comprises the steps of acquiring a data transmission terminal from the embedded device, displaying data of each hitting action and corresponding hitting time through a terminal gui interface, enabling a user or a table tennis training person to see whether the actions are normal or not and the hitting force in real time or when the hitting person hits the ball back through a gui interface, displaying normal actions, and assisting teaching to improve the table tennis level of the student.

9. An intelligent table tennis training system is characterized by comprising embedded equipment, a binocular camera and a terminal;

the embedded device comprises a sensor for collecting accelerometer data, gyroscope data and pressure sensor data;

the binocular camera is used for obtaining videos of students playing table tennis in real time;

the terminal comprises a processor for data processing and a gui interface for data display;

the embedded equipment, the binocular camera and the terminal are connected through the wireless communication module to conduct data transmission.

10. The intelligent table tennis training system according to claim 9, wherein,

the sensor comprises a plurality of MEMS sensor modules, wherein the sensor is embedded in a table tennis handle and is powered by a micro battery, and a system-on-chip radio frequency transceiver circuit is used for data transmission;

the wireless communication module adopts a USR-WIFI232-S wireless transceiver chip, the module is connected with the processor through the UART, and the processor processes data.

Images