CN116701954A - A Method of Infrastructure Status Identification Based on IMU Data - Google Patents

Abstract

The embodiment of the invention provides an infrastructure state identification method based on IMU data, which comprises the following steps: acquiring IMU data acquired by an inertial measurement unit deployed on an infrastructure, and extracting a feature vector corresponding to the infrastructure from the IMU data by using a feature extractor; performing similarity calculation on the extracted feature vector corresponding to the infrastructure and the reference feature vector of various states preset for the infrastructure to obtain a similarity result; according to the similarity result, the current state of the infrastructure is determined, and according to the similarity result obtained by performing similarity calculation on the extracted feature vector of the infrastructure and the reference feature vector of the plurality of states preset for the infrastructure, the technical scheme of the embodiment of the invention can be widely applied to any infrastructure and identify any infrastructure state designated by a user in a mode of determining the state of the infrastructure, and has high universality and low cost.

Description

A Method of Infrastructure Status Identification Based on IMU Data

technical field

The present invention relates to the Internet of Things technology, in particular to the field of digital signal processing, and more specifically, to an infrastructure state identification method based on IMU data.

Background technique

With the rapid development of Internet of Things technology, the application of infrastructure status recognition can be seen everywhere in life, especially in the fields of smart home, industrial Internet, fire safety, and smart elderly care. However, infrastructure detection often requires customized embedded sensors and recognition algorithms for specific usage scenarios, resulting in increased costs and barriers between subdivided applications, which greatly hinders the popularization of smart sensing technology. In this context, many sensing technologies have been proposed, including contact and non-contact methods.

Non-contact sensing technology uses radar, RFID, UWB and other radio frequency technologies to collect echo signals carrying the state characteristics of the detected object through the backscattering and multipath effects of radio waves on the detected object to identify the state of the object. However, this method is often limited by scenario specificity, high cost, and difficulty in deployment. It is only implemented in specific scenarios or cases, and has not been applied commercially on a large scale.

The contact sensing technology is mostly based on the inertial navigation unit, because it adopts MEMS (called MicroElectromechanical System, Micro Electromechanical System) technology, which has the characteristics of low power consumption and low cost. Inertial navigation units usually integrate accelerometers, gyroscopes, and magnetometers, which can characterize the attitude and direction changes of the detected facilities, and are widely used to detect object states, such as home state detection, facility attitude detection, vehicle impact detection, fall detection, etc. . Its detection technology is mostly based on sensor data for attitude and direction calculations, and state recognition through changes in intensity or direction, but such calculations are complex and inefficient. With the development of machine learning technology, more universal machine learning methods have gradually evolved, from the traditional extraction of semantic features such as pose direction to the iterative training of parameterized models based on specific scenarios to extract non-semantic features for state identify. However, this kind of state recognition based on machine learning relies on the labeled data of specific tasks, and the transferability between tasks is poor. If it covers multi-task scenarios, the number of parameters is too large and the calculation cost is too large.

Based on the above analysis, the existing methods of detecting the status of facilities through non-contact sensing technology are often limited by scene specificity, resulting in low versatility, high cost, and difficult deployment. Calculation of pose and direction for state recognition is computationally complex. Machine learning-based state recognition relies on task-specific labeled data. Poor transferability further leads to low versatility and excessive computational costs during the transfer process. Therefore, there is an urgent need for an infrastructure status recognition system with strong versatility and low computational cost.

Contents of the invention

Therefore, the object of the present invention is to overcome the above-mentioned defects in the prior art, and provide a method for identifying infrastructure status based on IMU data.

The purpose of the present invention is achieved by the following technical solutions:

According to a first aspect of the present invention, there is provided a method for identifying infrastructure status based on IMU data, the method comprising: acquiring IMU data collected by an inertial measurement unit deployed on the infrastructure, and using a feature extractor to extract Extract the eigenvector corresponding to the infrastructure; calculate the similarity between the extracted eigenvector corresponding to the infrastructure and the reference eigenvectors of various states preset for the infrastructure, and obtain the similarity result; according to the similarity result, Determine the current state of your infrastructure.

In some embodiments of the present invention, the feature extractor is obtained through the following training methods: obtaining a constructed neural network classification model, which includes a feature extractor and a classifier; obtaining a first training set, the first training set includes a plurality of The first sample data and the label corresponding to each first sample data, the first sample data is the IMU human body motion data collected when the human body performs a certain action, and the label indicates the human body action category corresponding to the first sample data Utilize the first training set to train the neural network classification model to recognize the human action category according to the first sample data, wherein the training feature extractor performs feature extraction on the first sample data of the first training set to obtain a feature vector, and train The classifier outputs the human action category according to the extracted feature vector, determines the first loss based on the additive angle margin loss function based on the label of the first sample data and the output human action category, and updates the feature extractor and classifier according to the first loss parameters.

In some embodiments of the present invention, the feature extractor includes a first branch network, a second branch network, a cascade layer, a Dropout layer and a fully connected layer, wherein the first branch network includes a multi-layer convolutional neural network, The second branch network includes a gated recurrent neural network, and the extracted feature vector is obtained by using the multi-layer convolutional neural network in the first branch network to perform convolution processing on the IMU data to extract the first feature; The gated recurrent neural network in the branch network processes the IMU data and extracts the second feature; uses the cascade layer to concatenate the first feature and the second feature to obtain the first hidden layer feature vector; uses the dropout layer to extract the second feature A feature vector of the first hidden layer is randomly deactivated to obtain a feature vector of the second hidden layer; a fully connected layer is used to process the feature vector of the second hidden layer to obtain an extracted feature vector.

In some embodiments of the present invention, the feature extractor is obtained through the following training methods: obtaining a built self-encoder, which includes a feature extractor and a decoder, and both the feature extractor and the decoder include a multi-layer convolutional neural network ; Obtain a second training set, the second training set includes a plurality of second sample data, each second sample data is to obtain a section of IMU data collected by an inertial measurement unit deployed in the infrastructure according to a predetermined time interval; use the second training set The training feature extractor performs feature encoding on the second sample data to obtain a feature vector, decodes the feature vector through a decoder to obtain decoded data, determines the second loss based on the mean square error loss function according to the second sample data and the decoded data, and The parameters of the feature extractor and decoder are updated according to the second loss.

In some embodiments of the present invention, the feature extractor includes a low-pass filter and a statistical feature module, wherein the extracted feature vector is obtained by using a low-pass filter to analyze the inertial measurement unit deployed on the infrastructure The collected IMU data is filtered to obtain the IMU signal features; the statistical feature module is used to statistically process the IMU data collected by the inertial measurement unit deployed on the infrastructure to obtain statistical features. Zero rate, skewness, kurtosis, and upper quartile; among them, the feature extractor splices the IMU signal features and statistical features to obtain the extracted feature vector.

In some embodiments of the present invention, for different types of infrastructure, there are respectively preset reference feature vectors corresponding to the type of infrastructure in two states or more than two states, each type of infrastructure is a door, a window , robotic arm, drawer, bed, curtain, water pipe, printer, exercise equipment, vent, toilet.

In some embodiments of the present invention, the current state of the infrastructure is determined in the following manner: each time the extracted feature vector corresponding to the infrastructure is compared with the reference feature vector of one of the preset states Compare the similarity result between them with the preset threshold, and when the similarity result is greater than the preset threshold, the preset state corresponding to the similarity result is taken as the current state of the infrastructure; or the extracted feature corresponding to the infrastructure The similarity calculation is performed between the vector and the reference feature vector of each state in the various preset states for it, and multiple similarity results are obtained. From the multiple similarity results, the state with the highest similarity result corresponding to the preset state is selected as the infrastructure current status.

In some embodiments of the present invention, the reference feature vector of each state in the preset multiple states includes the reference IMU signal feature and the reference statistical feature of the corresponding state, and the method of determining the state of the infrastructure is as follows: using dynamic The time warping similarity matching algorithm calculates the similarity calculation between the IMU signal features obtained by filtering and the reference IMU signal features of the preset corresponding state, and obtains the signal feature similarity result; compares the obtained statistical features with the preset corresponding state benchmark Statistical features are used to perform similarity calculations to obtain statistical feature similarity results; when both the signal feature similarity results and statistical feature similarity results are greater than the preset threshold, the state of the infrastructure is determined to be the preset corresponding state.

According to the second aspect of the present invention, there is provided an infrastructure state recognition system based on IMU data, the system is deployed on various infrastructures, and the system includes: a sensor data acquisition module for acquiring inertial measurements deployed on infrastructures The IMU data of the infrastructure collected by the unit; the microcontroller module, used to determine the current state of the infrastructure based on the IMU data of the infrastructure according to any one of the methods in the first aspect of the present invention; A communication module for transmitting the current infrastructure status determined by the microcontroller module to the user for status monitoring.

According to a third aspect of the present invention, there is provided an infrastructure state recognition system based on IMU data, the system is deployed on an intelligent gateway, a local host, a fog computing service platform or a cloud computing service platform, and sensor data acquisition is arranged in the infrastructure module and communication module, the sensor data acquisition module is used to obtain the IMU data collected by the inertial measurement unit deployed on the infrastructure, and the communication module is used to transmit the collected infrastructure IMU data to the system, and the system includes: a microcontroller module , for determining the current state of the infrastructure based on the infrastructure IMU data according to any one of the methods in the first aspect of the present invention; the communication module is used for transmitting the current infrastructure state determined by the microcontroller module to users for condition monitoring.

According to a fourth aspect of the present invention, there is provided an infrastructure state identification and alarm device based on IMU data, comprising: a sensor data acquisition module for acquiring IMU data collected by an inertial measurement unit deployed on the infrastructure; a microcontroller module , for determining the current state of the infrastructure based on the IMU data of the infrastructure according to any one of the methods in the first aspect of the present invention; the early warning module is used for judging the state of the infrastructure according to the current state of the infrastructure Whether the state is abnormal, and an alarm will be issued when the state is abnormal.

According to a fifth aspect of the present invention, there is provided an electronic device, comprising: one or more processors; and a memory, wherein the memory is used to store executable instructions; the one or more processors are configured to execute the The instructions are executable to implement the steps of any one of the methods described in the first aspect of the present invention.

Compared with the prior art, the present invention has the advantages of:

In the infrastructure state identification method of the present invention, firstly, the inertial measurement unit can be quickly deployed on the infrastructure and the deployment cost is low, and the IMU data collected by the inertial measurement unit on the infrastructure is directly obtained, and the feature vector of the IMU data is extracted to be used in the When it is necessary to monitor and control the state of the corresponding infrastructure, it can quickly identify its state; secondly, the reference feature vectors corresponding to the various states preset for the infrastructure are not limited by the specificity of the scene facilities and have high flexibility. The extracted The similarity calculation between the eigenvectors of the infrastructure and its preset reference eigenvectors is low in calculation cost; finally, according to the similarity results, the degree of similarity between the extracted eigenvectors corresponding to the infrastructure and the reference eigenvectors can be directly determined, involving There are few parameters and the current state of the infrastructure can be determined efficiently and quickly, and it is easy to realize real-time monitoring of the infrastructure. The state identification method of the present invention can be widely used in any infrastructure and identify any infrastructure state specified by the user, with high universality and low cost, such as being applied to various household appliances and industrial production machinery and equipment to realize monitoring and control Corresponds to the state of the infrastructure.

Description of drawings

Embodiments of the present invention will be further described below with reference to the accompanying drawings, wherein:

Fig. 1 is a kind of flow chart of infrastructure state recognition method based on IMU data according to one embodiment of the present invention;

Fig. 2 is a schematic structural diagram of a first feature extractor according to an embodiment of the present invention;

Fig. 3 is the schematic diagram of the process of feature extraction and classification of the input IMU human body action data according to the training neural network classification model according to one embodiment of the present invention;

Fig. 4 is a structural schematic diagram of an autoencoder comprising a feature extractor and a decoder constructed according to an embodiment of the present invention;

5 is a schematic diagram of the structure and principle of a third feature extractor according to an embodiment of the present invention;

6 is a schematic diagram of an IMU signal characteristic sequence and a reference IMU signal characteristic sequence obtained by aligned filtering according to an embodiment of the present invention;

FIG. 7 is a schematic diagram of a deployment scheme of an infrastructure state identification system according to an embodiment of the present invention;

FIG. 8 is a schematic diagram of a deployment scheme of an infrastructure state identification system according to another embodiment of the present invention;

Fig. 9 is a schematic diagram of a deployment scheme of an infrastructure state identification system according to another embodiment of the present invention.

Detailed ways

In order to make the object, technical solution and advantages of the present invention clearer, the present invention will be further described in detail below through specific embodiments in conjunction with the accompanying drawings. It should be understood that the specific embodiments described here are only used to explain the present invention, not to limit the present invention.

As mentioned in the background technology section, the existing method of detecting the state of facilities through non-contact sensing technology has low versatility, high cost, and is not easy to deploy, while the method of detecting the state of facilities using contact sensing technology is complex in calculation and difficult to implement .

In order to solve the above problems, the inventor objectively analyzed the existing method of detecting the state of facilities through non-contact sensing technology, which is often limited by scene specificity, which leads to low versatility, high cost, and difficult deployment. Using contact sensor technology, That is, the inertial measurement unit is deployed on the infrastructure, and the IMU (Inertial Measurement Unit) data corresponding to the state change of the infrastructure is collected through the inertial measurement unit to reduce the difficulty and cost of deployment. Further, the existing contact sensing technology detects the state of the facility Among them, the calculation of attitude and direction based on sensor data for state recognition is complex, and state recognition based on machine learning depends on the labeled data of specific tasks. Poor transferability further leads to low versatility and excessive computing costs during the transfer process. , therefore, the inventor pre-sets the reference feature vectors of various states for the infrastructure, and at the same time, uses the feature extractor to extract the corresponding feature vector of the infrastructure from the IMU data collected in the infrastructure, and compares the extracted feature vector with The similarity calculation is performed on the reference eigenvectors corresponding to the preset multiple states of the infrastructure, so as to match and determine the current state of the infrastructure, which greatly improves the computational efficiency and reduces the computational cost.

Based on the above analysis, according to an embodiment of the present invention, a method for identifying infrastructure status based on IMU data is proposed. Referring to FIG. 1, in step S1, the IMU data collected by the inertial measurement unit deployed on the infrastructure is obtained, using The feature extractor extracts the feature vector corresponding to the infrastructure from the IMU data, the inertial measurement unit can be quickly deployed on the infrastructure with low deployment cost, directly obtains the IMU data collected by the inertial measurement unit on the infrastructure, and extracts the IMU The eigenvector of the data is used to quickly identify the state of the corresponding infrastructure when it is necessary to monitor and control the state; in step S2, the extracted eigenvector corresponding to the infrastructure is compared with the reference characteristics The similarity calculation is performed on the vectors to obtain similarity results. The present invention presets reference feature vectors of various states for corresponding infrastructures, that is, different infrastructures can preset reference feature vectors corresponding to the possible states of the facility, and is not affected by Limited to the specificity of the scene facilities, the flexibility is high, and the similarity calculation between the extracted feature vector of the infrastructure and the preset reference feature vector is performed, and the calculation cost is low; in step S3, according to the similarity result, determine the basis The current state of the facility, the degree of similarity between the extracted feature vector and the reference feature vector can be directly determined through the similarity results, involving few parameters and efficiently and quickly determining the state of the infrastructure, which is easy to implement. real time monitoring. The state identification method of this embodiment can be widely used in any infrastructure and identify any infrastructure state specified by the user, with high universality and low cost, such as being applied to various household appliances and industrial production machinery and equipment to realize monitoring Control the state of the corresponding infrastructure.

According to an embodiment of the present invention, in the infrastructure state identification method of the present invention, an inertial measurement unit is installed and deployed on the infrastructure by means of attachment, pasting, embedding, etc. The inertial measurement unit is also called an inertial navigation unit, and an inertial measurement unit Generally, three sensors of accelerometer, gyroscope, and magnetometer are integrated. Therefore, the IMU data collected by the inertial measurement unit generally includes accelerometer data, gyroscope data, and magnetometer data. The types of data included in the IMU data are different. The IMU data collected by the inertial measurement unit in the present invention may include the three kinds of data of accelerometer data, gyroscope data and magnetometer data or any one of these three kinds of data or any data. Two types of data, such as accelerometer data only, or only accelerometer data and gyroscope data, for different types of infrastructure.

According to one embodiment of the present invention, a method for identifying the state of infrastructure based on IMU data is provided. After the method deploys an inertial measurement unit on the infrastructure and the inertial measurement unit collects the IMU data when the infrastructure executes a certain state, The state of the infrastructure is identified based on the IMU data of the infrastructure collected by the inertial measurement unit, and the identification method includes the following steps S1, S2 and S3. In order to better understand the present invention, each step will be described in detail below in conjunction with specific embodiments.

In step S1, the IMU data collected by the inertial measurement unit deployed on the infrastructure is obtained, and a feature vector corresponding to the infrastructure is extracted from the IMU data by a feature extractor.

According to an embodiment of the present invention, after obtaining the IMU data collected by the inertial measurement unit deployed on the infrastructure, any one of the following three examples of feature extractors can be used to realize the extraction from the IMU data The eigenvector corresponding to this infrastructure. Three examples of feature extractors are described below:

example one

Referring to Fig. 2, Fig. 2 is the structural representation of the first kind of feature extractor, and this feature extractor comprises the first branch network, the second branch network, cascading layer, Dropout layer (i.e. randomly discarding layer) and fully connected layer, wherein , the first branch network includes a multi-layer convolutional neural network. In the present invention, the first branch network is provided with a three-layer convolutional neural network, and the second branch network includes a gated recurrent neural network. The first type of feature extractor extracting the feature vector corresponding to the infrastructure from the IMU data includes: using a multi-layer convolutional neural network in the first branch network to perform convolution processing on the IMU data to extract the first feature; using The gated cyclic neural network in the second branch network processes the IMU data and extracts the second feature; uses the cascade layer to concatenate the first feature and the second feature to obtain the first hidden layer feature vector; uses the Dropout layer The feature vector of the first hidden layer is randomly inactivated to obtain the feature vector of the second hidden layer; the feature vector of the second hidden layer is processed by the fully connected layer to obtain the extracted feature vector. The feature extractor of the present invention includes a first branch network and a second branch network, that is, it adopts a dual-stream heterogeneous structure to ensure the accuracy of feature extraction. Layer feature vectors are randomly deactivated to ensure the generalization of the feature extractor.

According to an embodiment of the present invention, the feature extractor is a trained feature extractor, and the trained feature extractor is obtained through the training of the following steps a1, a2 and a3:

In step a1, the constructed neural network classification model is obtained, which includes a feature extractor and a classifier. Wherein, the structure of the feature extractor in the neural network classification model is the structure of the first feature extractor in FIG. 2 .

In step a2, the first training set is obtained. The first training set includes a plurality of first sample data and labels corresponding to each first sample data. The first sample data is collected when the human body performs a certain action The IMU human body motion data, the label indicates the human body motion category corresponding to the first sample data.

Since the infrastructure is designed for specific state change scenarios, and the data sets of infrastructure in the network are scarce, and at the same time, the quality of the data sets is limited, the present invention uses the atomic action data sets of the human body to train the neural network classification model to learn the continuous posture of the sensor. Characterization of the process of change. Therefore, according to an embodiment of the present invention, the first training set constructed uses the UTD-MHAD data set (multimodal data set for human action recognition), which contains 27 different atomic actions of the human body. It basically includes the attitude changes of inertial sensors (ie accelerometers, gyroscopes, and magnetometers) in all directions. The present invention adopts the existing data set, and does not need a large amount of data collection for specific infrastructure and its state, which reduces costs and improves efficiency; in addition, human body movement data is inherently more complicated than the state changes of infrastructure. The neural network classification model trained with human action data has better generalization, and the trained feature extractor obtained from the trained neural network classification model can better extract feature vectors from IMU data , and perform state recognition based on the extracted feature vectors, which greatly improves the accuracy of infrastructure state recognition.

In step a3, use the first training set to train the neural network classification model to identify the human body action category according to the first sample data, see Figure 3, Figure 3 is to train the neural network classification model to perform feature extraction on the input IMU human action data and a schematic diagram of the classification process, wherein the training feature extractor performs feature extraction on the first sample data (i.e. IMU human action data) of the first training set to obtain a feature vector, and the training classifier outputs the human action category according to the extracted feature vector , based on the additive angular margin loss function, the first loss is determined according to the label of the first sample data and the output human action category, and the parameters of the feature extractor and the classifier are updated according to the first loss.

According to an embodiment of the present invention, the calculation formula of the additive angular margin loss function Additive Angular Margin Loss is as follows:

Among them, loss _Arcface represents the first loss, N is the total amount of the first sample data participating in the training, s represents the scaling factor, which is used to control the length of the feature vector, θ _yi is the current first sample data in the FC (fully connected layer ) output feature vector and y _i correspond to the angle between the FC (full connection layer) weight vector, y _i represents the label of the i-th first sample data, m is the margin, and the margin is used to expand the training In order to increase the generalization ability of the neural network classification model for unseen sample data, n represents, θ _j represents the feature vector output by the data corresponding to category j in FC (full connection layer) and category j corresponds to FC ( The angle between the weight vectors of the fully connected layer), and j represents the category other than the human action category. Until the preset number of iterations is reached or the constructed neural network classification model converges, the training is stopped and the parameters of the feature extractor and classifier are updated to obtain a trained feature extractor.

Example two

Referring to Fig. 4, Fig. 4 is the structure diagram that comprises the self-encoder of feature extractor and decoder of construction, and its feature extractor and decoder all comprise multi-layer convolutional neural network, feature extractor and decoder among the present invention Both are equipped with a three-layer convolutional neural network. After the feature extractor in the autoencoder is trained, the trained second feature extractor is used to extract the corresponding features of the infrastructure from the IMU data. vector. Among them, when training the feature extractor in the autoencoder, the IMU data is input into the feature extractor for feature encoding to obtain the feature vector, and the decoder decodes the feature vector to obtain the data restored by the decoder, and according to the original input The data and the decoded data use the mean square error loss function to calculate the second loss and update the autoencoder parameters. Wherein, the feature extractor in the self-encoder can be obtained through the training methods of the following steps b1, b2 and b3:

In step b1, the constructed autoencoder is obtained, which includes a feature extractor and a decoder, both of which include a multi-layer convolutional neural network.

In step b2, a second training set is obtained, the second training set includes a plurality of second sample data, and each second sample data is a piece of IMU data collected by an inertial measurement unit deployed in the infrastructure at predetermined time intervals.

In step b3, use the second training set to train the feature extractor to perform feature encoding on the second sample data to obtain the feature vector, and decode the feature vector through the decoder to obtain the decoded data, based on the mean square error loss function according to the second sample The data and the decoded data determine a second loss, and the parameters of the feature extractor and decoder are updated according to the second loss.

According to an embodiment of the present invention, the calculation formula of the mean square error loss function Mean Squared Error Loss is as follows:

Among them, loss _MSE represents the second loss, N is the total amount of the second sample data participating in the training, y _i is the i-th IMU data segment input, Restores the output decoded data for the decoder. Until the preset number of iterations is reached or the constructed self-encoder converges, the training is stopped to update the parameters of the feature extractor and the decoder to obtain the trained second feature extractor.

Example three

Referring to Fig. 5, Fig. 5 is a schematic diagram of the structural principle of the third feature extractor, the feature extractor includes a low-pass filter and a statistical feature module, wherein the extracted feature vector is obtained by using the low-pass filter to The IMU data collected by the inertial measurement unit deployed on the infrastructure is filtered to obtain the IMU signal characteristics; the statistical feature module is used to statistically process the IMU data collected by the inertial measurement unit deployed on the infrastructure to obtain statistical features. Features include mean, variance, zero-crossing rate, skewness, kurtosis, and upper quartile; among them, the feature extractor splices IMU signal features and statistical features to obtain the extracted feature vector. The third feature extractor in the present invention splices the IMU signal features obtained by filtering the original IMU data and the statistical features obtained by performing feature statistics on the original IMU data, and obtains feature vectors that include various aspects of the original IMU data Data characteristics to ensure the correctness of infrastructure status identification.

In step S2, the similarity calculation is performed between the extracted feature vector corresponding to the infrastructure and the reference feature vectors of various states preset for the infrastructure to obtain a similarity result.

According to an embodiment of the present invention, for different types of infrastructure, there are respectively preset reference feature vectors corresponding to the type of infrastructure in two states or more than two states, and each type of infrastructure is a door, a window, One of robotic arms, drawers, beds, curtains, plumbing, printers, gym equipment, vents, toilets. Schematically, the preset states of doors, windows, mechanical arms, drawers, beds, curtains, water pipes, printers, fitness equipment, vents, and toilets in the above infrastructures include open and closed states, such as the The state is the door open and closed state, the state of the robotic arm is its power on and off state, and the state of the curtain is the unfolded and contracted state. Each state corresponding to each type of infrastructure can be preset according to user needs, that is, the user configures the reference feature vector of the corresponding state for the infrastructure. It should be understood that, according to the needs of different implementers, some preset states of the infrastructure can be customized and adjusted, rather than necessarily following the preset states given in the foregoing exemplary embodiments. For example, the corresponding preset state of the mechanical arm includes raising, lowering, moving left, right, moving forward, moving backward, and rotating; the corresponding preset state of the bed includes moving left, moving right, raising the bed support, The lowered state of the bed support; the corresponding preset state of the printer includes the printing and scanning states; the corresponding preset state of the toilet includes the flushing, lid opening, and lid closing states.

According to an embodiment of the present invention, execution actions in the same state of infrastructure may have deviations in speed and degree of completion, and there may also be deviations between execution actions in different states. Therefore, whether the user wants to monitor several states or monitor the same state Multiple reference feature vectors can be entered for different execution actions. When it is necessary to detect different completion degrees and different completion speeds of the same state, it is necessary to input multiple reference feature vectors, and the multiple base station feature vectors all represent the same state. For example, the corresponding preset state of the robotic arm includes raising and lowering states, and when the raising and lowering are completed, its speed can be fast or slow. Therefore, different speeds can be preset for the raised state of the infrastructure. A reference eigenvector, the reference eigenvector at each speed indicates that the manipulator is in a raised state, further improving the versatility and accuracy of the infrastructure state identification method.

According to an embodiment of the present invention, the method of calculating the similarity between the extracted feature vector corresponding to the infrastructure and the reference feature vectors of various states preset for the infrastructure includes: Method 1, the extracted infrastructure Calculate the similarity between the corresponding eigenvector and the reference eigenvector of one of the various states preset for the infrastructure, and obtain a similarity result. According to the similarity result, determine the extracted eigenvector and the state’s Whether the reference eigenvectors are similar, if they are similar, determine that the current state of the infrastructure is the state corresponding to the reference eigenvectors similar to its eigenvectors, and stop comparing the extracted eigenvectors with the reference eigenvectors of other states in various states Similarity calculation, if not similar, continue to calculate the similarity between the extracted feature vector and the reference feature vectors of other states in various states, until the reference feature vector similar to the feature vector corresponding to the infrastructure is matched, to Determine the status of your infrastructure. This method calculates and compares the extracted eigenvectors corresponding to the infrastructure with the preset reference eigenvectors of various states one by one. When the two eigenvectors are similar, the calculation and comparison are stopped to reduce the amount of calculation to a certain extent. Improve recognition efficiency. Method 2: Calculate the similarity between the extracted eigenvector corresponding to the infrastructure and the reference eigenvector of each of the various states preset for the infrastructure to obtain multiple similarity results, each similarity result Indicates the degree of similarity between the extracted eigenvectors corresponding to the infrastructure and the reference eigenvectors preset for each state of the infrastructure, and select the one corresponding to the highest similarity of the extracted eigenvectors from the obtained multiple similarity results A benchmark eigenvector to determine the state of the infrastructure. This method calculates and compares the extracted eigenvectors with the preset reference eigenvectors of all states, selects the reference eigenvector corresponding to the maximum similarity, and improves the recognition accuracy of the infrastructure state. It should be understood that the above two methods are only for illustration, and the user can choose according to actual needs.

According to an embodiment of the present invention, the similarity calculation adopts calculation methods including but not limited to calculating the cosine distance, the Euclidean distance, the cosine distance, the Euclidean distance, Manhattan distance, Pearson correlation coefficient or Spearman correlation coefficient, etc. According to the calculated cosine distance, Euclidean distance, Manhattan distance, Pearson correlation coefficient or Spearman correlation coefficient, the similarity result indicating the similarity of two feature vectors is obtained ; Similarity calculation also includes using methods such as dynamic time warping similarity matching algorithm to directly perform similarity matching calculation on sensor data, such as the third feature extractor given in the above example 3 to extract the infrastructure from the collected IMU data The corresponding feature vector, the extracted feature vector includes IMU signal features and statistical features, then the reference feature vector corresponding to each state in the various states preset for the infrastructure includes the reference IMU signal features and reference statistics of the corresponding state Features, use the dynamic time warping similarity matching algorithm to calculate the similarity between the IMU signal features extracted in Example 3 and the reference IMU signal features, and use the cosine similarity algorithm to perform similarity between the statistical features extracted in Example 3 and the reference statistical features degree calculation. When the user requires a low degree of accuracy for the detection action, when either of the two similarity results reaches the threshold condition set by the user, the current infrastructure status can be determined.

In step S3, the current state of the infrastructure is determined according to the similarity result.

According to an embodiment of the present invention, based on the method of similarity calculation in step S2, the method of determining the current state of the infrastructure is: each time the extracted feature vector corresponding to the infrastructure is compared with the preset multiple The similarity result between the reference feature vectors of one of the states is compared with the preset threshold, and when the similarity result is greater than the preset threshold, the preset state corresponding to the similarity result is taken as the current state of the infrastructure. Among them, when the similarity result is greater than the preset threshold, the extracted feature vector is considered similar to the reference feature vector of the corresponding state, otherwise the two feature vectors are not similar. For example, the preset state for the infrastructure includes state 1, state For state two and state three, calculate and compare the extracted eigenvector corresponding to the infrastructure with the preset reference eigenvectors of multiple states one by one, that is, to perform cosine similarity calculation between the extracted eigenvector and the reference eigenvector of state 1, The obtained similarity result is 0.8, and the preset threshold is set to 0.95, then the extracted feature vector is not similar to the reference feature vector of state 1, continue to calculate the similarity between the extracted feature vector and the reference feature vector of state 2, and obtain similarity If the degree result is 0.96, if 0.96 is greater than the preset threshold value of 0.95, it means that the extracted feature vector is similar to the reference feature vector of state 2, stop calculating and comparing with the reference feature vector corresponding to state 3, and use state 2 as the current infrastructure state. Among them, the preset threshold comes from the empirical setting of the best recognition effect for the current state of the infrastructure. In actual use, the user can set the threshold to match different recognition accuracy. A high threshold requires a high degree of matching between the actual infrastructure state and the expected state, and high flexibility.

According to an embodiment of the present invention, the way to determine the current state of the infrastructure based on the second method of similarity calculation in step S2 is: compare the extracted feature vector corresponding to the infrastructure with various preset states The similarity calculation is performed on the reference feature vector of each state in , and multiple similarity results are obtained. From the multiple similarity results, the state with the highest similarity result corresponding to the preset is selected as the current state of the infrastructure. Taking the feature vector obtained by the first type of feature extractor given in Example 1 above as an example, the feature vector corresponding to the infrastructure obtained by the first type of feature extractor is compared with each of the various states preset for the infrastructure The cosine similarity calculation is performed on the reference eigenvectors of the infrastructure. The preset states include a total of 6 types, and the extracted eigenvectors corresponding to the infrastructure are calculated with the reference eigenvectors of each of the 6 preset states. Then get 6 similarity results, respectively 0.5, 0.6, 0.4, 0.97, 0.91, 0.8, select the benchmark eigenvector corresponding to the highest similarity result 0.97, and determine the current state of the infrastructure as the state indicated by the benchmark eigenvector. It should be understood that the present invention does not limit the method of determining the current state of the infrastructure, and the user can choose a corresponding method to determine the current state of the infrastructure according to requirements.

According to an embodiment of the present invention, the feature vector obtained by the third feature extractor given in the above example three is taken as an example to illustrate the similarity result obtained by the dynamic time warping similarity matching algorithm, and determine the basis based on the similarity result The way of the state of the facility comprises the following steps c1, c2 and c3:

In step c1, the similarity calculation is performed between the IMU signal features obtained by calculating and filtering by using the dynamic time warping similarity matching algorithm and the reference IMU signal features of the preset corresponding state, to obtain a signal feature similarity result.

Schematically, see Figure 6, Figure 6 is a schematic diagram of the IMU signal feature sequence and the reference IMU signal feature sequence obtained by aligned filtering, the sensor sequence A generated for the target action entered by the user, and the sensor sequence B collected at a certain time , when dynamic time warping is used for similarity calculation, it is assumed that the lengths of the IMU signal feature sequence Q and the reference IMU signal feature sequence C are n and m respectively, Q=q ₁ ,…,q _i ,…,q _n ,q ₁ is the first point of sequence Q, q _i is the ith point of sequence Q, q _n is the nth point of sequence Q, C=c ₁ ,...,c _j ,...,c _m , c ₁ is the sequence The first point of C, c _j is the jth point of sequence C, c _m is the mth point of sequence C, construct an n*m matrix to align the two sequences, for example, each matrix element (i ,j) indicates the aligned points q _i and c _j , d(q _i ,c _j ) indicates the distance between the two points q _i and c _j corresponding to the matrix element (i,j), that is, each point of the sequence Q The similarity between each point and sequence C is generally calculated by Euclidean distance, d(q _i ,c _j )=(q _i -c _j ) ² , the smaller the distance, the higher the similarity result, and the larger the distance Indicates the lower the similarity result. Based on the Euclidean distance corresponding to each element in the calculated matrix, the dynamic programming idea is used to find a continuous monotonous shortest cumulative distance path in the matrix. The sum of the paths is the dynamic time warping distance of the two feature sequences. The dynamic time warping distance is Indicates the final calculated signal feature similarity result.

In step c2, the similarity calculation is performed between the obtained statistical features and the preset benchmark statistical features of the corresponding state to obtain a statistical feature similarity result. The cosine distance between the statistical features obtained by cosine similarity calculation and the preset benchmark statistical features of the corresponding state, the larger the cosine distance, the higher the statistical feature similarity result, and the smaller the cosine distance, the better the statistical feature similarity result Low.

In step c3, when both the signal feature similarity result and the statistical feature similarity result are greater than a preset threshold, it is determined that the state of the infrastructure is a preset corresponding state. It is set to determine the current infrastructure state only when the signal feature similarity results and statistical feature similarity results reach the threshold conditions set by the user, so as to improve the accuracy of state identification.

According to one embodiment of the present invention, an infrastructure state recognition system based on IMU data is provided, the system is deployed on various infrastructures, and the system includes: a sensor data acquisition module for acquiring inertial measurements deployed on infrastructures The IMU data of the infrastructure collected by the unit; the microcontroller module, used to determine the current state of the infrastructure based on the IMU data of the infrastructure according to the infrastructure state identification method of the above-mentioned embodiment; communication Module for transmitting the current infrastructure status determined by the microcontroller module to the user for condition monitoring.

Schematically, refer to FIG. 7, which is a schematic diagram of a deployment scheme of an infrastructure state recognition system, and the embedded chip used is a microcontroller Nrf52833 with a Bluetooth communication function. The embedded chip includes three modules: sensor data acquisition module, microcontroller module, communication module. Deploy the algorithm (that is, the infrastructure state identification method in the above-mentioned embodiment) at the edge end, that is, deploy it in the embedded chip, and the embedded chip is packaged into a certain product form and fixed on the infrastructure by pasting or other fixed methods, such as Doors, windows, industrial production robotic arms, etc. After the user configures the reference eigenvectors of various states for the corresponding infrastructure according to the state he needs to monitor, when monitoring the state of the infrastructure, the sensor data acquisition module obtains the IMU data collected by the inertial measurement unit on the infrastructure, micro The controller module performs feature extraction and similarity calculation on the IMU data to determine the current state of the infrastructure, the communication module transmits the current state of the infrastructure, and the communication module also transmits bidirectional control commands. The two-way control instructions mainly include remote call and configuration instructions, which are used to implement the server (such as cloud service) to call the functions in the embedded chip. Embedded chips can also perform remote calls for data operations in servers (such as cloud services), remind users of infrastructure anomalies through cloud services, and issue alarms for dangerous states of infrastructure perceived by the edge. Among them, the model parameter quantization technology is used to complete the deployment of the model (the model is the feature extractor in the present invention), that is, a certain strategy is adopted to convert the parameters of the model from floating-point values to fixed-point values so as to reduce the parameter space of the model, such as the model The quantization measurement adopts the maximum and minimum value quantization method min-max, which is an offline quantization method implemented after model training, and maps the floating-point parameter range of the model to the 8-bit fixed-point parameter range of -128-128 layer by layer, thereby compressing the above model To about 100KB in size, it is deployed on the microcontroller module of the Nrf52833 embedded chip.

According to another embodiment of the present invention, there is provided an infrastructure state recognition system based on IMU data, the system is deployed on an intelligent gateway, a local host, a fog computing service platform or a cloud computing service platform, and sensor data is provided in the infrastructure The acquisition module and the communication module, the sensor data acquisition module is used to obtain the IMU data collected by the inertial measurement unit deployed on the infrastructure, the communication module is used to transmit the collected infrastructure IMU data to the system, and the system includes: a microcontroller A module for determining the current state of the infrastructure based on the infrastructure IMU data according to the infrastructure state identification method of the above-mentioned embodiment; a communication module for transmitting the current infrastructure state determined by the microcontroller module to users for condition monitoring.

Schematically, refer to FIG. 8, which is a schematic diagram of another deployment scheme of an infrastructure state recognition system, in which the algorithm (that is, the infrastructure state recognition method in the above-mentioned embodiment) is deployed on the fog end server (for the area server or a cluster of regional server nodes) or within a microcontroller module in a cloud server (cloud service). In this example, an embedded chip is installed on the infrastructure where the inertial measurement unit is deployed. For industrial production of manipulators, etc., the embedded chip includes two modules: a sensor data acquisition module and a communication module. After the user configures reference feature vectors of various states for the corresponding infrastructure according to the state it needs to monitor, the IMU data is processed by the inertial measurement unit. Acquisition, when monitoring the status of the infrastructure, the sensor data acquisition module acquires IMU data in real time and transmits it to the fog end server or cloud server, and the microcontroller module in the corresponding server performs feature extraction and similarity calculation to obtain the infrastructure status , and transmit the determined current infrastructure status to the user through the communication module in the server for status monitoring.

Schematically, refer to FIG. 9, which is a schematic diagram of another deployment scheme of an infrastructure state identification system. The algorithm (that is, the infrastructure state identification method in the above-mentioned embodiment) is deployed in the microcontroller module in the gateway, and through the edge The embedded chip at the end obtains the IMU data collected by the inertial measurement unit and uploads it to the gateway, and the gateway performs local real-time infrastructure status identification. The edge embedded chip deployed in this example includes two modules: sensor data acquisition module and communication module. The embedded development board is packaged into a certain product form and fixed on the infrastructure by pasting or other fixing methods, such as doors, windows, industrial production robotic arms, etc. The IMU data is collected by the inertial measurement unit. When monitoring the status of the infrastructure, the sensor data acquisition module acquires the IMU data in real time and transmits it to the gateway. The microcontroller module in the corresponding gateway performs feature extraction and similarity calculation to obtain the basis The status of the facility is uploaded to the cloud server or fed back to the embedded chip, and the determined current infrastructure status is transmitted to the user through the cloud server for status monitoring.

According to one embodiment of the present invention, there is provided an infrastructure state identification and alarm device based on IMU data, including: a sensor data acquisition module for acquiring IMU data collected by an inertial measurement unit deployed on the infrastructure; a microcontroller module , used to determine the current state of the infrastructure based on the IMU data of the infrastructure according to the infrastructure state identification method of the above-mentioned embodiment; the early warning module is used to judge the state of the infrastructure according to the current state of the infrastructure Whether it is abnormal, and an alarm will be issued when the state is abnormal.

It should be noted that although the steps are described above in a specific order, it does not mean that the steps must be performed in the above specific order. In fact, some of these steps can be performed concurrently, or even change the order, as long as it can be realized The required functions are sufficient.

The present invention can be a system, method and/or computer program product. A computer program product may include a computer readable storage medium having computer readable program instructions thereon for causing a processor to implement various aspects of the present invention.

A computer readable storage medium may be a tangible device that holds and stores instructions for use by an instruction execution device. A computer readable storage medium may include, for example, but is not limited to, electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), or flash memory), static random access memory (SRAM), compact disc read only memory (CD-ROM), digital versatile disc (DVD), memory stick, floppy disk, mechanically encoded device, such as a printer with instructions stored thereon A hole card or a raised structure in a groove, and any suitable combination of the above.

Having described various embodiments of the present invention, the foregoing description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and alterations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principle of each embodiment, practical application or technical improvement in the market, or to enable other ordinary skilled in the art to understand each embodiment disclosed herein.

Claims (13)

1. A method for identifying infrastructure state based on IMU data, characterized in that the method comprises: Obtain the IMU data collected by the inertial measurement unit deployed on the infrastructure, and use the feature extractor to extract the feature vector corresponding to the infrastructure from the IMU data; Calculate the similarity between the extracted eigenvectors corresponding to the infrastructure and the reference eigenvectors of various states preset for the infrastructure to obtain similarity results; Based on the similarity results, determine the current state of the infrastructure. 2. method according to claim 1, is characterized in that, described feature extractor obtains by following training method: Obtain the constructed neural network classification model, which includes feature extractors and classifiers; Obtain the first training set, the first training set includes a plurality of first sample data and the label corresponding to each first sample data, the first sample data is the IMU human body action data collected when the human body performs a certain action , the label indicates the human action category corresponding to the first sample data; Utilize the first training set to train the neural network classification model to identify the human action category according to the first sample data, wherein the training feature extractor performs feature extraction on the first sample data of the first training set to obtain a feature vector, and train the classification The human body action category is output by the device according to the extracted feature vector, the first loss is determined based on the label of the first sample data and the output human action category based on the additive angular margin loss function, and the feature extractor and the classifier are updated according to the first loss parameter. 3. The method according to claim 2, wherein the feature extractor comprises a first branch network, a second branch network, a cascade layer, a Dropout layer and a fully connected layer, wherein the first branch network comprises multiple layer convolutional neural network, the second branch network includes a gated recurrent neural network, and the extracted feature vector is obtained by: Using the multi-layer convolutional neural network in the first branch network to perform convolution processing on the IMU data, and extract the first feature; The IMU data is processed by the gated recurrent neural network in the second branch network, and the second feature is extracted; Concatenating the first feature and the second feature by using a cascade layer to obtain a feature vector of the first hidden layer; Using the Dropout layer to perform random inactivation processing on the feature vector of the first hidden layer to obtain the feature vector of the second hidden layer; The feature vector of the second hidden layer is processed by the fully connected layer to obtain the extracted feature vector. 4. method according to claim 1, is characterized in that, described feature extractor obtains by following training method: Obtain the built autoencoder, which includes a feature extractor and a decoder, both of which include a multi-layer convolutional neural network; Obtain a second training set, the second training set includes a plurality of second sample data, and each second sample data is a section of IMU data collected by an inertial measurement unit deployed in the infrastructure according to a predetermined time interval; Use the second training set to train the feature extractor to perform feature encoding on the second sample data to obtain the feature vector, decode the feature vector through the decoder to obtain the decoded data, and determine based on the mean square error loss function based on the second sample data and the decoded data second loss, and update the parameters of the feature extractor and decoder according to the second loss. 5. method according to claim 1, it is characterized in that, described feature extractor comprises low-pass filter and statistical feature module, wherein, obtain the feature vector of extraction in the following way: Use a low-pass filter to filter the IMU data collected by the inertial measurement unit deployed on the infrastructure to obtain the IMU signal characteristics; The IMU data collected by the inertial measurement unit deployed on the infrastructure is statistically processed by the statistical feature module to obtain statistical features, where the statistical features include mean, variance, zero-crossing rate, skewness, kurtosis, and upper quartile site; Among them, the feature extractor splices the IMU signal features and statistical features to obtain the extracted feature vector. 6. The method according to claim 1, characterized in that, for different types of infrastructure, there are respectively preset reference feature vectors corresponding to the type of infrastructure in two states or more than two states, each Infrastructure is one of doors, windows, robotic arms, drawers, beds, curtains, plumbing, printers, gym equipment, vents, toilets. 7. The method according to claim 1, wherein the current state of the infrastructure is determined in the following manner: Each time the similarity result between the extracted feature vector corresponding to the infrastructure and the reference feature vector of one of the preset states is compared with the preset threshold, when the similarity result is greater than the preset threshold , the similarity result corresponds to the preset state as the current state of the infrastructure; or Calculate the similarity between the extracted eigenvector corresponding to the infrastructure and the reference eigenvector of each state in the preset multiple states to obtain multiple similarity results, and select the similarity result from the multiple similarity results The highest corresponds to the preset state as the current state of the infrastructure. 8. The method according to claim 5, wherein the reference feature vector of each state in the preset multiple states includes a reference IMU signal feature and a reference statistical feature of the corresponding state, to determine the state of the infrastructure The way is as follows: Using the dynamic time warping similarity matching algorithm to calculate and filter the IMU signal features and the preset reference IMU signal features of the corresponding state to perform similarity calculations to obtain the signal feature similarity results; Calculate the similarity between the obtained statistical features and the preset benchmark statistical features of the corresponding state, and obtain the statistical feature similarity results; When both the signal feature similarity result and the statistical feature similarity result are greater than a preset threshold, it is determined that the state of the infrastructure is a preset corresponding state. 9. An infrastructure state recognition system based on IMU data, said system is deployed on each infrastructure, it is characterized in that the system includes: The sensor data collection module is used to obtain the IMU data of the infrastructure collected by the inertial measurement unit deployed on the infrastructure; A microcontroller module, configured to determine the current state of the infrastructure based on the IMU data of the infrastructure according to the method according to any one of claims 1-8; A communication module for transmitting the current infrastructure status determined by the microcontroller module to the user for status monitoring. 10. An infrastructure state recognition system based on IMU data, characterized in that the system is deployed on an intelligent gateway, a local host, a fog computing service platform or a cloud computing service platform, and the infrastructure is provided with a sensor data acquisition module and a communication module, the sensor data acquisition module is used to obtain the IMU data collected by the inertial measurement unit deployed on the infrastructure, and the communication module is used to transmit the collected infrastructure IMU data to the system, and the system includes: A microcontroller module, configured to determine the current state of the infrastructure based on the infrastructure IMU data according to the method according to any one of claims 1-8; A communication module for transmitting the current infrastructure status determined by the microcontroller module to the user for status monitoring. 11. An infrastructure state identification alarm device based on IMU data, characterized in that, comprising: The sensor data acquisition module is used to acquire the IMU data collected by the inertial measurement unit deployed on the infrastructure; A microcontroller module, configured to determine the current state of the infrastructure based on the IMU data of the infrastructure according to the method according to any one of claims 1-8; The early warning module is used to judge whether the state of the infrastructure is abnormal according to the current state of the infrastructure, and to give an alarm when the state is abnormal. 12. A computer-readable storage medium, wherein a computer program is stored thereon, and the computer program can be executed by a processor to implement the steps of the method according to any one of claims 1-8. 13. An electronic device, characterized in that it comprises: one or more processors; and memory, where the memory is used to store executable instructions; The one or more processors are configured to implement the steps of the method of any one of claims 1 to 8 by executing the executable instructions.