CN116741204A - An abnormal sound detection method based on hierarchical metadata information constraints - Google Patents

Abstract

The invention discloses an abnormal sound detection method based on hierarchical metadata information constraint, which comprises the steps of converting an audio waveform of sound to be detected into Log-Mel spectral frequency characteristics, and then inputting the Log-Mel spectral frequency characteristics into a pre-trained characteristic extractor to obtain advanced audio characteristicsComputing advanced audio featuresEach attribute group center c corresponding to a machine ID of a sound to be detected _m Selecting the minimum value as an anomaly score A, wherein M is the number of attribute groups under the corresponding machine ID, and judging that the sound to be detected is different when A is larger than a given threshold valueConstant sound; the attribute group center c _m The average value of the advanced audio characteristics obtained by a pre-trained characteristic extractor for the audio fragments of the training set; the invention designs a metadata information tree structure, fully utilizes metadata information to extract finer characteristics, effectively improves the performance of the abnormal sound detection system, and solves the problems of insufficient performance and low reliability of detection results of the existing industrial abnormal sound detection method under domain offset.

Description

An abnormal sound detection method based on hierarchical metadata information constraints

Technical field

The invention belongs to the technical field of abnormal sound detection methods, and relates to an abnormal sound detection method based on hierarchical metadata information constraints.

Background technique

Anomalous Sound Detection (ASD) aims to automatically detect abnormal sounds emitted by the target machine when it is in an abnormal condition.

With the continuous development of deep learning technology in the audio field, autoencoder architecture is widely used in unsupervised abnormal sound detection. Existing methods usually use the log-Mel Spectrogram of the sound when the mechanical equipment is operating normally as a feature input to train the autoencoder model, and output the log-Mel Spectrogram of the same number of frames as the input. The graph is used as reconstructed features to detect abnormal sounds. However, during the training process, the autoencoder only contains the constraints of normal sounds and does not contain information about abnormal sounds. If the features obtained by training are not well applicable to abnormal sounds, the effectiveness of the autoencoder reconstruction method will be reduced. restricted.

During the training process of neural networks, sufficient label data is needed for constraints, but obtaining abnormal data itself is a challenge in the field of industrial abnormal sound detection. To solve this problem, existing self-supervised methods convert unsupervised models into supervised models to better learn compact representations of normal data. One of the auxiliary tasks of self-supervised classification learns fine features of normal sounds by training a classifier to predict the machine ID of each machine and distinguishing them from abnormal sounds by discerning the machine IDs of accompanying audio data as labels. . If the classifier incorrectly classifies the machine ID of the sound data, it is considered an anomaly. However, due to the domain shift problem in the real world, the main features of training and test data sometimes do not have similar distributions, and the performance of abnormal sound detection is often limited in practice. For example, changes in machine operating conditions or noise types may cause acoustic properties to differ between the source domain (training) and the target domain (detection), so a model trained using sounds from the source domain may incorrectly identify anomalies in the target domain sound.

The self-supervised classification scheme adopts machine IDs as auxiliary labels of audio files for feature learning because each machine ID represents a specific type of domain offset. However, in addition to machine type and machine ID, sounds are also associated with various properties, such as how fast the machine is operating. Therefore, changes in attribute values are also one of the causes of domain drift and are crucial to affecting domain drift. Using machine ID alone may not be sufficient to obtain features that help characterize domain shifts. Self-supervised machine attribute classification considers the impact of industrial machine attributes on acoustic characteristics and uses it as an auxiliary label for self-supervised feature learning. However, this system does not make full use of the metadata information associated with the audio file itself, so the extracted feature representations are not refined enough.

Contents of the invention

In view of the above-mentioned existing technologies, the technical problem to be solved by the present invention is to provide an abnormal sound detection method based on hierarchical metadata information constraints, to solve the problem that the metadata information attached to the audio files of the machine is not fully utilized, and self-supervision under domain offset The feature representation extracted by this method is not precise enough.

In order to solve the above technical problems, the present invention provides an abnormal sound detection method based on hierarchical metadata information constraints, including:

Convert the audio waveform of the sound to be detected into Log-Mel spectral frequency features, and then input it into the pre-trained feature extractor to obtain advanced audio features Calculate advanced audio features/> The Mahalanobis distance of the center c _m of each attribute group corresponding to the machine ID of the sound to be detected is selected, and the minimum value is selected as the anomaly score A,/> M is the number of attribute groups under the corresponding machine ID. When A is greater than the given threshold, the sound to be detected is determined to be an abnormal sound;

The attribute group center c _m is the average value of the high-level audio features obtained by the pre-trained feature extractor of the training set audio clips;

The training process of the feature extractor includes:

Select a set of normal sound clips from the machine as a training set;

Divide the audio clips with the same attributes and attribute values in the training set audio clips corresponding to each machine ID into an attribute group, and each machine ID and corresponding attribute group constitute hierarchical metadata information;

Convert the audio waveform of the training set into Log-Mel spectral frequency features and send them to the feature extractor to obtain the low-level features f _l and high-level features f _h of the audio;

The low-level features f _l and high-level features f _h are respectively sent to the machine ID classifier C _ID and the machine AG classifier C _AG to obtain the predicted values of the machine ID auxiliary label by the machine ID classifier C _ID respectively. and the predicted value of machine AG auxiliary label by machine AG classifier C _AG /> C _ID (·) represents the machine ID classifier, C _AG (·) represents the machine attribute classifier;

Use the total cross-entropy loss function L _Total to train the feature extractor to obtain the trained feature extractor, L _Total = λL _ID + (1-λ)L _AG , λ is the set weight parameter, and L _ID represents the predicted value and the loss function of the difference value between the machine ID tag l _ID in the hierarchical metadata information, L _AG represents the predicted value/> and the loss function of the difference value of the machine attribute group label l _AG in the hierarchical metadata information.

Further, the feature extractor includes a deep network with an attention mechanism and a two-dimensional convolution layer. The low-level features f _l are extracted through the deep network with the attention mechanism, and then the high-level features are extracted through the two-dimensional convolution layer. _fh .

Furthermore, the deep network with attention mechanism is MobileFaceNet.

Beneficial effects of the present invention:

1) In view of the problem that existing self-supervision methods do not fully utilize metadata information, the present invention designs a metadata information tree structure to fully utilize metadata information to extract more refined features;

2) The hierarchical metadata information constraint method designed by the present invention can effectively improve the performance of the abnormal sound detection system and solve the problem of insufficient performance of existing industrial abnormal sound detection methods under domain offset and low credibility of detection results.

Description of drawings

Figure 1 is the overall technical roadmap of the present invention;

Figure 2 is a hierarchical metadata information structure diagram constructed in the present invention;

Figure 3 is the backbone network structure built based on MobileFaceNet.

Detailed ways

The present invention will be further described below in conjunction with the accompanying drawings and examples.

The present invention is an abnormal sound detection method based on hierarchical metadata information constraints, and obtains the hierarchical relationship between machine IDs and attributes with the help of the constructed metadata information tree structure of each machine. The technical solution of the present invention is implemented as follows:

Step 1: Extract the Log-Mel spectrum characteristics of the machine audio in the data set, and use the hierarchical relationship between machine IDs and attributes to build a hierarchical metadata information structure;

Step 2: Feed the Log-Mel spectral frequency features into the feature extractor to obtain low-level and high-level feature representations. Use the obtained feature representation and hierarchical metadata information structure to train the model;

Step 3: Use the trained model to use anomaly detection based on the attribute group center on the test sound to determine whether the sound is abnormal.

Further, the step 1 specifically includes:

Step 101: Use fast Fourier transform and Mel filter bank to convert the audio waveform into Log-Mel spectral frequency characteristics;

Step 102: Construct an attribute group (AttributeGroup, AG) using different attributes corresponding to the audio clips and their corresponding values;

Step 103: Use the machine ID and the attribute group obtained in step 102 to construct a hierarchical metadata information structure, and obtain the machine ID auxiliary label l _ID and the machine AG auxiliary label l _AG . The hierarchical relationship between the machine ID and machine attributes is used for learning and domain bias. transfer-related features.

Further, the step 2 specifically includes:

Step 201: Send the Log-Mel spectral frequency features obtained in step 101 to the feature extractor to obtain low-level and high-level feature representations of the audio;

Step 202: Send the low-level and high-level feature representations obtained in step 201 to the machine ID classifier C _ID and machine AG classifier C _AG respectively to obtain the predicted values of the machine ID auxiliary label and machine AG by the machine ID classifier C _ID respectively. The predicted value of classifier C _AG for machine AG auxiliary label, its formula is as follows: and/>

Step 203: Send the predicted values of the machine ID and machine AG obtained in step 202 to the training total cross-entropy loss function L _Total to train the HMIC-based abnormal sound detection model. The formula is as follows: and/>

Further, the step 3 specifically includes:

Step 301: Calculate the center point of each AG of the normal sound during the training process, that is, the Attribute Group Center (AGC). The center of the AG is the average value of the audio features learned by the model in each attribute group. To detect anomalies in test data with domain offset, the calculation formula is as follows:

Among them, c _m represents the center of the m-th attribute group, Represents the high-level audio features obtained from the model of the nth training audio clip, n∈[1,N].

Step 302: Use the Mahalanobis distance between the feature representation of the test data and the AGC as an anomaly score to measure the audio feature representation of the test data The similarity between the center c _m of each attribute group and the Mahalanobis distance is a distance measure that takes into account the correlation and covariance matrix and can be used to evaluate the similarity between unknown samples and training data. The closer the Mahalanobis distance between the test sample and the AGC is, the more likely the unknown sound is to be normal data. The further the Mahalanobis distance to the AGC is, the more likely the unknown sound is to be abnormal data. The calculation formula is as follows:/>

Among them, A represents the anomaly score, Σ ^-1 is the inverse matrix of the covariance matrix Σ, and Σ is obtained from the features of all audio segments under the segment with the same m-th attribute group.

Examples are given below in conjunction with specific parameters.

Combined with Figure 1, the present invention provides an abnormal sound detection method based on hierarchical metadata information constraints, which specifically includes the following steps:

step 1:

Fast Fourier transform and Mel filter bank are used to extract Log-Mel spectral features from the original sound signal, where the frame size is set to 1024, the frame jump size is 512, and the number of Mel filter banks is 128. For 10 seconds of audio, at least 313 frames of log-mel spectrogram features need to be generated, so the input log-mel spectrogram feature dimension is 313×128.

A hierarchical metadata information structure is then constructed. Figure 2 is a hierarchical metadata information structure diagram constructed in the present invention, further utilizing the hierarchical relationship between machine IDs and attributes. Since the audio clips under each machine ID may have some attributes with different values, these attributes and corresponding values are grouped together into an AG under the machine ID, so that multiple attributes with different attribute values for each machine ID can be obtained. AG, get the machine ID auxiliary label l _ID and the machine AG auxiliary label l _AG . The metadata information tree structure of each machine type is constructed, where the machine ID is the node of the information tree and AG is the leaf of the information tree. This hierarchical relationship is used as a constraint in self-supervised learning to obtain a more refined audio feature representation, where the machine ID characterizes the domain shift type for low-level feature learning, and the attribute group utilizes the acoustic properties of the domain shift for high-level feature learning.

Taking the machine type as a toy car as an example, the toy car with machine ID 00 contains 4 attributes with different attribute values, where Car represents the car model, Spd represents the speed of the car, Mic represents the number of collecting microphones, and Noise represents the environmental noise level, for example Car's values are A1, C2, etc., while Noise's values are 1, 2, etc. By grouping these attributes and corresponding values, the number of AGs for the toy car with machine ID 00 is 11, and the number of AGs for all machine IDs of the machine type toy car is 44. A total of 250 AGs under 42 machine IDs for all machine types (i.e., seven machine types, each with six machine IDs) are therefore used to build the hierarchical relationship between machine IDs and attributes.

Step 2:

Send the Log-Mel spectrum and frequency features obtained in step 1 to the feature extractor to obtain low-level feature representation f _l and high-level feature representation f _h :

f _l =F(X)

The low-level feature representation f _l extracts the high-level feature representation f _h through the two-dimensional convolutional layer in the backbone network:

f _h =Conv2D(f _l )

Among them, F(·) represents the feature extractor in the backbone network, and Conv2D(·) represents the 2D convolution layer in the backbone network. Figure 3 shows the backbone network structure built based on MobileFaceNet, which is implemented with the help of feature extractor modules and 2D convolutional layers. In addition, the structure of the backbone network is not limited to the above structure and can be replaced by a deep network layer with an attention mechanism.

The hierarchical relationship between machine ID and machine attributes obtained in step 1 is used to learn features related to domain offset. It uses low-level machine ID to constrain low-level feature learning, and it uses high-level machine AG to constrain high-level feature learning:

The low-level and high-level feature representations obtained in step 2 are respectively sent to the machine ID classifier C _ID and the machine AG classifier C _AG to obtain the predicted values of the machine ID auxiliary label by the machine ID classifier C _ID and the machine AG classifier C _AG respectively. Predicted value for machine AG-assisted tags:

The predicted values of machine ID and machine AG obtained in step 2 are respectively sent to the training total cross-entropy loss function L _Total to train the abnormal sound detection model based on HMIC:

L _Total =λL _ID +(1-λ)L _AG ;

where λ is a weight parameter chosen empirically during training. The weight parameter λ is adjusted for each machine type and is the same for all machine IDs of the machine type. Among them, L _ID and L _ID are:

Among them, CE(·) represents the cross-entropy (Cross-Entropy, CE) loss function

Step 3:

The average value of the audio features learned for each attribute group is calculated as the AGC to evaluate the test sound. The AGC contains the acoustic characteristics of normal sounds related to the domain shift, so it can be used to measure the anomalies of the test samples with domain shift. The anomaly score is the Mahalanobis distance between the audio feature representation of the detected sound and the AGC.

Assume that there are N training audio clips under the m-th attribute group, and the labels are Where m∈[1,M], M is the number of attribute groups under the corresponding machine ID.

Among them, c _m represents the center of the m-th attribute group, Represents the high-level audio features obtained from the model of the nth training audio clip, n∈[1,N].

Mahalanobis distance to measure audio feature representation of detected sounds Similarity to the center c _m of each attribute group:

Among them, A represents the anomaly score, Σ ^-1 is the inverse matrix of the covariance matrix Σ, and Σ is obtained from the features of all audio segments under the same segment of the m-th attribute group.

The present invention provides an abnormal sound detection method based on hierarchical metadata information constraints, which effectively solves the defect of insufficient performance of existing abnormal sound detection methods under domain offset. Table 1 and Table 2 show the traditional methods and applications. The present invention provides a post-strategy method. AUC value scale comparison of abnormal sound detection performance uses the commonly used abnormality detection evaluation index AUC to reflect the overall test performance in the source domain and target domain. Table 3 shows the abnormal sound detection performance of the traditional method and the method after applying the strategy provided by the present invention. Partial AUC (pAUC) index comparison uses pAUC to illustrate the overall test performance of the noise detection method under low false alarm rate, thus reflecting the practicality of the method.

The abnormal sound detection method based on hierarchical metadata information constraints provided by the present invention far exceeds the abnormality detection performance of existing traditional methods and the abnormality detection performance under low false alarm rate, and achieves better AUC performance in multiple fields and stronger The pAUC performance strongly illustrates that the strategy of the present invention can excellently complete abnormal sound detection under domain offset, bringing better performance.

Table 1

Table 2

table 3

Claims (3)

1. An abnormal sound detection method based on hierarchical metadata information constraint is characterized by comprising the following steps:

the audio waveform of the sound to be detected is converted into Log-Mel spectral frequency characteristics, and then the Log-Mel spectral frequency characteristics are input into a pre-trained characteristic extractor to obtain advanced audio characteristicsComputing advanced audio features->Each attribute group center c corresponding to a machine ID of a sound to be detected _m Selecting the minimum value as the abnormality score A, < >>M is the number of attribute groups under the corresponding machine ID, and when A is greater than a given threshold value, the sound to be detected is judged to be abnormal sound;

the attribute group center c _m For training set audio clip channelAn average value of the advanced audio features obtained by the feature extractor trained in advance;

the training process of the feature extractor comprises the following steps:

selecting a group of normal sound clips of the machine as a training set;

dividing the audio clips with the same attribute and attribute value in the training set audio clips corresponding to each machine ID into an attribute group, wherein each machine ID and the corresponding attribute group form hierarchical metadata information;

converting the training set audio waveform into Log-Mel spectral frequency characteristics and sending the characteristics to a characteristic extractor to obtain low-level characteristics f of audio _l And advanced feature f _h ；

Will lower the level of features f _l And advanced feature f _h Respectively sent into a machine ID classifier C _ID And machine AG classifier C _AG Respectively obtain the machine ID classifier C _ID Predictive value for machine ID assisted tagsAnd machine AG classifier C _AG Predictive value for machine AG auxiliary tag +.>C _ID (. Cndot.) represents a machine ID classifier, C _AG (. Cndot.) represents a machine attribute classifier;

using a total cross entropy loss function L _Total Training the feature extractor to obtain a trained feature extractor, L _Total ＝λL _ID +(1-λ)L _AG Lambda is a set weight parameter, L _ID Representing predicted valuesAnd machine ID tag/in hierarchical metadata information _ID A loss function of the difference value between L _AG Representing predicted value +.>And hierarchical metadataMachine attribute group label in information _AG A loss function of the difference value of (2).

2. The abnormal sound detection method based on hierarchical metadata information constraint of claim 1, wherein: the feature extractor comprises a depth network with an attention mechanism and a two-dimensional convolution layer, and extracts low-level features f through the depth network with the attention mechanism _l Then extracting by a two-dimensional convolution layer to obtain an advanced feature f _h 。

3. The abnormal sound detection method based on hierarchical metadata information constraint of claim 1, wherein: the deep network with the attention mechanism is MobileFaceNet.