CN112069963A - Low-quality dangerous target detection method and system based on non-invasive neural signals - Google Patents

Abstract

The invention discloses a method and system for detecting a low-quality dangerous target based on a non-invasive neural signal, the method comprising: shooting a monitoring video or image by an unmanned aerial vehicle, and collecting EEG signals when monitoring personnel watch the video or image; The EEG signals are preprocessed and feature extracted, and the extracted features are input into the classifier to detect whether dangerous objects appear in the returned video or image. The method and system for detecting low-quality dangerous targets based on non-invasive neural signals provided by the present invention can quickly and accurately identify low-quality dangerous targets in complex videos or images.

Description

A method and system for low-quality dangerous target detection based on non-invasive neural signals

technical field

The invention relates to the fields of cognitive neuroscience and information technology, and more particularly to a method and system for detecting low-quality dangerous targets based on non-invasive neural signals.

Background technique

At present, drones equipped with high-definition digital cameras and cameras and GPS positioning systems can locate autonomous cruises along the set route, transmit and shoot images in real time, and monitor personnel can watch and control them simultaneously on the computer to identify whether there is danger in the monitoring area. Compared with the traditional manual patrol method, it is more flexible and fast, and it is very suitable for information collection on long-distance border lines.

Usually, two methods are generally used for video and image processing collected and returned by drones: manual recognition and machine recognition. Machine recognition processing efficiency is high, but it is not competent to recognize low-quality dangerous targets with characteristics such as camouflage, incompleteness, occlusion, and weakness. For images captured by drones, it is affected by many uncontrollable factors such as environment and weather. , the collected images and video data are often of low quality, which makes it difficult for machine recognition to meet the requirements of sensitivity, versatility and reliability; the human visual system is very good at analyzing scenes and recognizing objects, and manual recognition can easily find that the machine is difficult to recognize. The identified low-quality dangerous targets, but the manual identification processing speed is slow, easy to fatigue, it is difficult to deal with massive data processing, and it is difficult to meet the requirements.

It can be seen from the above analysis that the existing methods cannot meet the needs of fast and accurate detection and identification of low-quality dangerous targets in complex videos and images.

Therefore, how to quickly and accurately identify low-quality dangerous objects in complex videos or images is an urgent problem to be solved by those skilled in the art.

SUMMARY OF THE INVENTION

In view of this, the present invention provides a method and system for detecting low-quality dangerous targets based on non-invasive neural signals, which can quickly and accurately identify low-quality dangerous targets in complex videos or images.

In order to achieve the above object, the present invention adopts the following technical solutions:

A method for detecting low-quality dangerous objects based on non-invasive neural signals, comprising:

Collect EEG signals when monitoring personnel watch videos or images;

The EEG signals are preprocessed and feature extracted, and the extracted features are input into the classifier to detect whether dangerous objects appear in the returned video or image.

Preferably, the method further includes: photographing the monitoring area, wherein the photographed video or image is for monitoring personnel to watch.

Preferably, the classifier is retrained based on the new sample set.

Preferably, the preprocessing and feature extraction of the EEG signal specifically includes:

Perform independent component analysis on the EEG signal, filter out artifacts based on the pre-obtained approximate entropy threshold, and perform the inverse operation of independent component analysis on the independent components after eliminating the artifacts;

Down-sampling the EEG signal after the inverse operation of the independent component analysis, and obtain the best estimate of the EEG signal based on the Kalman smoothing method;

The best estimation based on EEG uses the directed transfer function for feature calculation, and uses PCA for feature compression to obtain the extracted features.

Preferably, inputting the extracted features into a classifier to detect whether a dangerous target appears in the returned video or image specifically includes:

The extracted features are fed into an LSTM analysis model to identify if dangerous objects appear in the returned video or image.

Preferably, it also includes: when no dangerous target is detected, saving the sample currently input into the LSTM analysis model as an additional new sample.

Preferably, the retraining of the classifier based on the new sample set specifically includes:

When the number of the additional new samples exceeds the threshold, the additional new samples and the pre-stored offline samples are used as a new sample set to retrain the classifier; wherein, the additional new samples are retrained after each retraining of the classifier Clear.

A low-quality dangerous target detection system based on non-invasive neural signals, comprising: an acquisition module, an EEG acquisition module and an EEG processing module;

The acquisition module photographs the monitoring area, wherein the photographed video or image is for monitoring personnel to watch;

The EEG acquisition module is used to collect EEG signals when monitoring personnel watch videos or images, and send the EEG signals to the EEG processing module;

The EEG processing module is used for preprocessing and feature extraction on the EEG signal, and inputting the extracted features into a classifier to detect whether a dangerous target appears in the returned video or image.

Preferably, the acquisition module includes an unmanned aerial vehicle.

Preferably, it also includes: an adaptive module;

The adaptation module is used to retrain the classifier based on the new sample set.

It can be seen from the above technical solutions that, compared with the prior art, the present invention provides a method and system for detecting low-quality dangerous targets based on non-invasive neural signals. Or the EEG signal generated when the image is generated, so as to determine whether a specific neural response is generated to determine whether a dangerous target appears.

The method and system for detecting a low-quality dangerous target based on a non-invasive neural signal provided by the present invention can detect the low-quality dangerous target in the target area accurately and quickly.

Description of drawings

In order to explain the embodiments of the present invention or the technical solutions in the prior art more clearly, the following briefly introduces the accompanying drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are only It is an embodiment of the present invention. For those of ordinary skill in the art, other drawings can also be obtained according to the provided drawings without creative work.

1 is a block diagram of a low-quality dangerous target detection system based on non-invasive neural signals of the present invention;

Fig. 2 is the location of the EEG acquisition electrode provided by the present invention;

3 is a schematic diagram of a flow chart of artifact filtering in the EEG processing module provided by the present invention;

4 is a schematic flowchart of a signal estimation method for nonlinear Kalman smoothing in an EEG processing module provided by the present invention;

FIG. 5 is a schematic flowchart of an EEG processing module provided by the present invention.

Detailed ways

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

To solve the problems existing in the prior art, it is necessary to combine manual identification and machine identification. When a dangerous target is detected by artificial recognition, the brain of the supervisor will generate a specific neural response, which can be detected by brain-computer interface (BCI) technology. BCI can establish a direct control and information exchange channel between computer and other external equipment and the human brain. It is an information exchange system that does not rely on peripheral nerves and muscle tissue. The neural signals used in BCI mainly include functional magnetic resonance imaging (fMRI), near-infrared spectroscopy (NIRS), magnetoencephalography (MEG) and electroencephalography (EEG). It is widely used (the neural signals used in the present invention are all EEG signals). The detection of the specific neural response has both the accuracy of manual recognition and the high processing efficiency of machine recognition.

The technical solution provided by the present invention combines manual recognition and machine recognition, and proposes a low-quality dangerous target detection method and system based on non-invasive neural signals. The EEG signals can be used to judge whether there is a dangerous target by judging whether a specific neural response is generated.

Referring to FIG. 1, an embodiment of the present invention discloses a method for detecting low-quality dangerous targets based on non-invasive neural signals, including:

Shoot the surveillance area, in which the captured video or image is for the monitoring personnel to watch; in the specific implementation, the target area is captured by the drone, and the video or image is transmitted in real time;

Collect the EEG signals when the monitoring personnel watch videos or images; it should be noted here that after the EEG signals are collected, the EEG signals also need to be amplified and converted to analog-digital processing. Electrical signals refer to EEG signals that have been processed by amplification and analog-to-digital conversion.

In addition, during the specific implementation, the EEG signals of the monitoring personnel are collected through the EEG electrodes placed on the scalp of the monitoring personnel's brain. For the specific location, please refer to FIG. 2 .

The EEG signals are preprocessed and feature extracted, and the extracted features are input into the classifier to detect whether dangerous objects appear in the returned video or image.

In order to further optimize the above technical solution, on the basis of the above-mentioned non-invasive neural signal-based low-quality dangerous target detection method, the method further includes: retraining the classifier based on a new sample set.

Referring to FIG. 3, in order to further optimize the above technical solution, the preprocessing and feature extraction of the EEG signal specifically includes:

Perform independent component analysis on the EEG signal, filter out artifacts based on the pre-obtained approximate entropy threshold, and perform the inverse operation of independent component analysis on the independent components after eliminating the artifacts;

Down-sampling the EEG signal after the inverse operation of the independent component analysis, and obtain the best estimate of the EEG signal based on the Kalman smoothing method;

The best estimation based on EEG uses the directed transfer function for feature calculation, and uses PCA for feature compression to obtain the extracted features.

In order to further optimize the above technical solution, the inputting the extracted features into the classifier to detect whether a dangerous target appears in the returned video or image specifically includes:

The extracted features are fed into an LSTM analysis model to identify if dangerous objects appear in the returned video or image.

In order to further optimize the above technical solution, the method further includes: when no dangerous target is detected, saving the sample currently input into the LSTM analysis model as an additional new sample.

In order to further optimize the above technical solution, the retraining of the classifier based on the new sample set specifically includes:

When the number of the additional new samples exceeds the threshold, the additional new samples and the pre-stored offline samples are used as a new sample set to retrain the classifier; wherein, the additional new samples are retrained after each retraining of the classifier Clear.

For example: the threshold is set to 100, when 100 additional new samples are saved, the additional new samples and pre-stored offline samples are combined as a new sample set to retrain the classifier.

In addition, the embodiment of the present invention also discloses a low-quality dangerous target detection system based on non-invasive neural signals, comprising: an acquisition module, an EEG acquisition module, and an EEG processing module;

The collection module shoots the monitoring area, wherein the captured video or image is for monitoring personnel to watch; preferably, the collection module includes an unmanned aerial vehicle.

The EEG acquisition module is used to collect EEG signals when monitoring personnel watch videos or images, and send the EEG signals to the EEG processing module;

The EEG processing module is used for preprocessing and feature extraction on the EEG signal, and inputting the extracted features into a classifier to detect whether a dangerous target appears in the returned video or image.

In order to further optimize the above technical solution, it also includes: an adaptive module;

The adaptation module is used to retrain the classifier based on the new sample set.

Among them, the EEG acquisition module uses an EEG acquisition instrument to collect EEG signals in real time, and performs amplification and analog-to-digital conversion, and transmits data to the EEG processing module through a data line. The total number of channels collected is 16. According to the "10-20 international standard leads", the EEG acquisition electrodes are placed on the F3, F4, Fz, C3, C4, Cz, P7, P3, P4, P8, Pz, O1, O2, Oz, T7, T8 positions, the reference electrodes are placed on the A11, A12 positions on the user's earlobe (each electrode position is shown in Figure 2). The sampling frequency is set to 1000Hz.

Among them, the EEG processing module is used for receiving EEG signals, and processing the EEG signals to detect whether the monitoring personnel have produced a specific neural response. The system processes the EEG signal with a window width of 1s and a step size of 0.1s. The processing process is shown in Figure 3, including: step 1, preprocessing the EEG signal, reducing the dimension, and filtering out noise; step 2, Features are extracted from the preprocessed data; step 3, the features are substituted into the classifier for classification.

Wherein, step 1 specifically includes:

1) Independent component analysis and approximate entropy

Various non-EEG signal artifacts (eye blinks, eye movements, EMG) and interfering signals (power frequency interference, mechanical effects caused by cable or electrode movement) in the collected EEG signals can reduce brain activity. The signal-to-noise ratio of an electrical signal. Aiming at the possible interference in the EEG signal, the signal is preprocessed.

The solid line in Figure 3 represents the data processing flow of the training process. First, the collected EEG data is analyzed by independent components, and the relevant unmixing matrix is obtained for independent component analysis of the EEG data during testing; The component labels corresponding to the independent components related to the trace, the approximate entropy is calculated based on the independent components and the component labels as the threshold value of automatic filtering, and the independent components related to the artifact are set to 0 to realize the filtering of the artifact, and then the independent components are carried out. The inverse of the analysis.

The dotted line in Figure 3 represents the actual test data processing flow. First, the EEG signal is analyzed by independent components, and the artifacts are filtered out based on the pre-obtained approximate entropy threshold, and the independent component analysis inverse operation is performed on the independent components after eliminating the artifacts.

2) Downsampling and Kalman smoothing

The EEG signal is downsampled to 200Hz to reduce the computational complexity, and after the artifacts are filtered out, the remaining EEG signal can be written as a state-space model. Since all data points of each data window are available, the Kalman smoothing method is used to obtain the best estimate of the EEG signal, which is compared with bandpass filters and recursive least squares (RLS) filters. has better estimation performance. Here, the nonlinear Kalman smoothing method is used for optimal estimation, see Figure 4 for details.

In Fig. 4, the solid line represents the data processing flow in the training process. First, the observation matrix H is obtained by calculating the EEG signal, and then the state transition matrix Φ is estimated by the AR model, and then the process noise Q and the observation noise R are estimated. Finally, the parameters obtained by H, Φ, Q, and R offline estimation are input into the nonlinear Kalman model for online calculation.

Step 2 specifically includes:

1) Calculation of directed transfer function

The first step in computing the directed transfer function is the estimation of the coefficient matrix of the multivariate autoregressive model (MVAR model). The multivariate autoregressive model can be expressed as:

Among them, x _{n, t} represents the sampling value of the nth channel of EEG data at time t, e is zero mean white noise, p is the order of the model, and a is the coefficient of the model. A multivariate autoregressive model can be understood as a model obtained by fitting the current data according to the data at p time points in the past. This formula can be expressed in matrix form:

Equation 2.2 can be further organized as:

In equations (2.2) and (2.3), X _t =[X _1,t ,...,X _n,t ] represents the n-dimensional EEG vector sampled by all channels at time t, and A _i is the coefficient matrix of X _ti , whose dimension is n×n, and E _t =[E _i,t ,...,E _n,t ] is a white noise vector. The task of estimating the coefficient matrix of a multivariate autoregressive model is to fit a model coefficient matrix of n×n×p dimensions:

A=[A ₁ ,...,A _i ,...,A _p ] (2.4)

The second step in calculating the directed transfer function is to use the matrix of model coefficients to calculate the directed transfer function matrix. First, use the Fourier transform (convolution property) to transform Equation (2.3) into the frequency domain and organize it, that is

H ^F = (A ^F ) ^-1 (2.6)

where ^AF represents the Fourier transform of the model coefficient matrix ^A , AF can be calculated by:

H ^F is the directed transfer function matrix, which is a three-dimensional matrix (channel × channel × frequency). The (k,l, ^f )th element h _k,l,f in HF is the directed transfer function from channel l to channel k at fHz, which can characterize the connectivity of channel l to channel k at fHz or The amount of information flowing from channel l to channel k at fHz.

The directed transfer function needs to be further normalized by:

where h ^* _k,l,f are the normalized directional transfer functions, correspondingly, the normalized directional transfer function matrix H ^* can be obtained as the original feature.

2) PCA

As mentioned earlier, the dimension of H ^* is very high, where a large number of original features may have a lot of similar information. In order to prevent feature redundancy and overfitting, the original features are first compressed by PCA. PCA projects the original feature vector into the principal component space, thereby replacing the original high-dimensional features with fewer features and retaining most of the information contained in the high-dimensional features.

Assuming that the input is an a×b-dimensional matrix X (sample×feature), the PCA algorithm first performs zero-mean standardization on each column element in the X matrix, that is:

where x _i,j are the elements of the i-th row and the j-th column of the matrix X, and μ _j and δ _j are the mean and standard deviation of the j-th column elements, respectively. Then calculate the covariance matrix C of the normalized X matrix.

Where X ^T represents the transpose of the X matrix, and trace() represents the trace of the matrix, that is, the sum of the diagonal elements. After that, perform singular value decomposition on the covariance matrix C, and arrange the eigenvectors and eigenvalues in descending order according to the eigenvalues, and the obtained eigenvector matrix is the projection matrix WPCA of PCA. Each eigenvalue and eigenvector corresponds to a principal component, and the contribution rate (information content) si of the first _i principal components can be calculated by the following formula:

where λ _k represents the k-th largest eigenvalue. On the premise that the contribution rate is close to 1, the minimum number of principal components i _min is used as the feature dimension after PCA compression. In the actual application of the PCA algorithm, by multiplying the original feature vector with the projection matrix W _pca , the first i _min values are taken out as the compressed feature.

Among them, step 3 is as follows:

The classifier uses a long short-term memory network (LSTM). LSTM mainly includes four parts: forgetting gate, input gate, output gate, and memory unit. The memory unit is responsible for storing useful information, and the three gating units are responsible for adding and deleting information to the memory unit.

The forgetting gate is responsible for controlling how much information of the memory unit at the previous moment can be retained to the current moment, and its formula is:

f _t =σ(W _f [h _t-1 , x _t ]+b _f ) (3.1)

Among them, the brackets indicate that the two vectors are connected and merged, W _f is the weight matrix of the forget gate, σ is the sigmoid function, and b _f is the bias term of the forget gate. h _t-1 is the output at the previous moment, and x _t is the input at the current moment.

The input gate is used to calculate which information is stored in the memory cell, and its formula is:

i _t =σ(W _i [h _t-1 , x _t ]+b _i ) (3.2)

Among them, the brackets indicate that two vectors are connected and merged, W _i , W _c are the weight matrix of the input gate, σ is the sigmoid function, tanh is the hyperbolic tangent function, and b _i , b _c are the bias terms of the input gate. h _t-1 is the output at the previous moment, and x _t is the input at the current moment.

The unit state at the current moment is the product of the forget gate input and the state at the previous moment plus the product of the two parts of the input gate:

For the output gate, the sigmod function is used to calculate what information needs to be output, and the output is obtained by multiplying the current unit state by the value of the tanh function. The formula is:

o _t =σ(W _o [h _t-1 , x _t ]+b ₀ ) (3.5)

h _t =o _t *tanh(c _t ) (3.6)

Among them, the brackets indicate that two vectors are connected and merged, W _o is the weight matrix of the output gate, σ is the sigmoid function, tanh is the hyperbolic tangent function, and b ₀ is the bias term of the input gate. c _t is the memory cell state at the current moment, x _t is the input at the current moment, and h _t is the output at the current moment.

The final classification result can be obtained by converting the current moment output through the classification layer.

Among them, the adaptive module is used to train the classifier to update the parameters of the classifier model, so that the model can adapt to changes in the state of the monitoring personnel. Specifically, the classifier is retrained by constructing an adaptive sample set. The adaptive sample set consists of two parts of samples: the first part of the sample is called the original sample, the original sample is the n-dimensional feature vector of the offline training classifier, and they are permanently stored in the adaptive sample set in advance. 100 normal samples and 50 dangerous target samples, and this part of the sample set is not updated; the second part of the samples is called additional new samples, and the additional new samples are also n-dimensional feature vectors, which are input from real-time to the classifier. Selected from the new n-dimensional feature vector of , additional new samples are used as normal samples. When the additional new samples are accumulated to a certain number (for example, the threshold of the present invention is set to 100), the classifier is retrained by using a new sample set composed of these two parts of the sample set, namely the adaptive sample set. After completing the retraining of the classifier each time, the additional new samples are cleared; after a period of time (the interval time set in the present invention is 10 seconds), the accumulation of additional new samples is restarted and the next adaptive process is completed.

The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same and similar parts between the various embodiments can be referred to each other. As for the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant part can be referred to the description of the method.

The above description of the disclosed embodiments enables any person skilled in the art to make or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Thus, the present invention is not intended to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims (10)

1. A method for detecting a low-quality risk target based on a non-invasive neural signal, comprising:

collecting electroencephalogram signals when monitoring personnel watch videos or images;

preprocessing and feature extraction are carried out on the electroencephalogram signals, and the extracted features are input into a classifier so as to detect whether dangerous targets appear in the returned video or image.

2. The method for detecting low-quality risk target based on non-invasive neural signals according to claim 1, further comprising: and shooting the monitored area, wherein the shot video or image is watched by the monitoring personnel.

3. The method of claim 1 or 2, wherein the classifier is retrained based on a new sample set.

4. The method for detecting the low-quality dangerous target based on the non-invasive neural signal as claimed in claim 3, wherein the pre-processing and feature extraction of the electroencephalogram signal specifically comprises:

carrying out independent component analysis on the electroencephalogram signals, filtering artifacts based on a pre-acquired approximate entropy threshold, and carrying out independent component analysis inverse operation on the independent components after the artifacts are eliminated;

performing down-sampling on the electroencephalogram signal subjected to the independent component analysis inverse operation, and acquiring the optimal estimation of the electroencephalogram signal based on a Kalman smoothing method;

and performing feature calculation by adopting a directed transfer function based on the optimal estimation of the electroencephalogram signals, and performing feature compression by using PCA (principal component analysis) to obtain extracted features.

5. The method according to claim 4, wherein the step of inputting the extracted features into a classifier to detect whether dangerous objects appear in the returned video or image comprises:

the extracted features are input into an LSTM analysis model to identify whether a dangerous target is present in the backtransmitted video or image.

6. The method for detecting low-quality risk target based on non-invasive neural signals according to claim 5, further comprising: when no dangerous target is detected, the sample currently input into the LSTM analysis model is saved as an additional new sample.

7. The method according to claim 6, wherein the retraining the classifier based on the new sample set specifically comprises:

when the number of the additional new samples exceeds a threshold value, the additional new samples and the prestored offline samples are used as a new sample set to retrain the classifier; wherein the additional new samples are purged after each retraining of the classifier.

8. A low-quality risk target detection system based on non-invasive neural signals, comprising: the device comprises an acquisition module, an electroencephalogram acquisition module and an electroencephalogram processing module;

the acquisition module shoots a monitored area, wherein the shot video or image is watched by monitoring personnel;

the electroencephalogram acquisition module is used for acquiring electroencephalogram signals when monitoring personnel watch videos or images and sending the electroencephalogram signals to the electroencephalogram processing module;

the electroencephalogram processing module is used for preprocessing the electroencephalogram signals and extracting features, and inputting the extracted features into a classifier so as to detect whether dangerous targets appear in the returned video or image.

9. The system of claim 8, wherein the acquisition module comprises a drone.

10. The system of claim 8, further comprising: an adaptive module;

the adaptation module is to retrain the classifier based on a new sample set.

Images