CN106934378A - A kind of dazzle light identifying system and method based on video depth study - Google Patents

Abstract

The invention discloses a car high beam recognition system and method based on video deep learning. The system includes the following two parts: the front part, which is used to realize the identification and processing of high beam violations, including sequentially connected road monitoring Equipment module, video processing and recognition module, recognition result processing module, and violation result database to be checked; the background part is used for video processing and deep learning of video, including key frame extraction algorithm, tagged database and deep learning module, with tagged The database is constructed by calling the key frame extraction algorithm for the original video data to extract key frames. The data in the labeled database is used for the training of the deep learning module. The trained deep learning module and the key frame extraction algorithm are used together for video processing and recognition. module call. The invention automatically analyzes and recognizes the monitoring video, ensures the completeness of law enforcement evidence, is similar to manual judgment, and has intelligence.

Description

A car high beam recognition system and method based on video deep learning

technical field

The invention relates to an automobile high beam recognition system, in particular to an automobile high beam recognition system and method based on video deep learning. It belongs to the field of intelligent transportation technology.

Background technique

Since the reform and opening up, my country's economy has achieved sustained, stable and rapid development, which has also improved the living standards of our people unprecedentedly, and more and more Chinese people have private vehicles. While the rapid growth of the number of private cars brings convenience to people's travel, it also makes the frequency of traffic accidents higher and higher.

There are many reasons for traffic accidents, many of which are caused by improper use of high beams. At present, the supervision of violations of high beam lights mainly relies on the capture of traffic police. Due to the limitation of police force and time, it is impossible to guarantee effective supervision of all violations of high beam lights. In addition, some high-beam capture systems developed in recent years all recognize the captured pictures, but these methods have certain limitations, as shown in: 1) The number of captured high-beam pictures is small and incoherent. The high-beam pictures are likely to be generated by the driver during normal use, and it is easy to be misjudged as using high-beams indiscriminately. Therefore, using pictures as evidence for law enforcement is not sufficient; 2) In order to obtain these pictures, it is often necessary to set up multiple additional capture devices at the same location , the cost is higher; 3) the original video monitoring equipment cannot be fully utilized, resulting in waste of resources.

Contents of the invention

The purpose of the present invention is to provide a car high beam recognition system based on video deep learning for overcoming the above-mentioned deficiencies in the prior art.

The present invention also provides a car high beam recognition method based on video deep learning corresponding to the above system.

To achieve the above object, the present invention adopts the following technical solutions:

A car high beam recognition system based on video deep learning, including the following two parts:

The foreground part is used to realize the identification and processing of high beam violations, including the road monitoring equipment module, video processing and identification module, recognition result processing module, and pending inspection results database;

The background part is used for video processing and deep learning of video, including key frame extraction algorithm, tagged database and deep learning module. The data in the label database is used for the training of the deep learning module, and the trained deep learning module and the key frame extraction algorithm are called by the video processing and recognition module.

As one of the preferred technical solutions, the key frame extraction algorithm is a cluster-based key frame extraction algorithm.

As one of the preferred technical solutions, the deep learning module is a deep learning module based on CNN+LSE (convolutional neural network+least square estimation).

A car high beam recognition method based on video deep learning corresponding to the above system, the specific steps are as follows:

(1) The road monitoring equipment module obtains the driving video data of the car and transmits it to the video processing and identification module;

(2) The video processing and recognition module calls the key frame extraction algorithm to extract the key frames of the video data, and then performs the grayscale operation, takes the grayscaled key frames as input, and calls the CNN+LSE based on the label database training The deep learning module obtains the output label of each key frame, including low beam, fog light or high beam, and assigns the label to the corresponding key frame image;

(3) The video data and the key frame with the label obtained in step (2) are used as the input of the recognition result processing module to determine whether the vehicle violates regulations, and the license plate recognition system is embedded in the recognition result processing module. When the target vehicle When there is a high beam violation, the license plate is extracted, the vehicle information is obtained, and the suspected violation video data is imported into the violation result database to be checked.

In step (2), the key frame extraction algorithm is as follows:

(2-1) Take the i-th segment V _i in the original video database, extract n frames at equal time intervals, and use F _i,j to name the frame at the j-th moment of the i-th segment video data, and set the key of the corresponding video data The frame sequence is expressed as {F _i,1 ,F _i,2 ,...,F _i,n }, where F _i , ₁ is the first frame, and F _i,n is the last frame; define the similarity between two adjacent frames The degree is the similarity of the histograms of two adjacent frames (that is, the histogram feature difference), and the predefined threshold δ controls the density of clustering; wherein, i, j and n are all integers;

(2-2) Select the first frame F _i,1 as the initial clustering center, and calculate the similarity between the frame F _i,j and the initial clustering center, if the value is less than δ, it is determined that the frame and the clustering center The distance between the center frames is too large, therefore, F _{i, j} cannot be added to the cluster; if the similarity between F _{i, j} and all cluster centers is less than δ, then F _{i, j} forms a new cluster, F _{i, j} is the new cluster center; otherwise, add the frame to the cluster with the largest similarity, so that the distance between the frame and the center of this cluster is the smallest;

(2-3) Repeat the operation of (2-2) to classify the n frames extracted from the original video data V _i into different clusters, then you can select the key frame: extract the distance from each cluster The nearest frame of the cluster center is used as the representative frame of this cluster, and the representative frames of all clusters constitute the key frame of the original video data V _i .

In step (2), the construction method of the tagged database is as follows:

A large amount of vehicle driving video data under the background of big data is used as the original video data, and the key frame extraction algorithm based on clustering is called on the original video data to extract the key frames, and the lighting type of the vehicle in the key frames is manually determined, and each key frame is added The label turns the original keyframe into labeled data. The label categories include: low beam, fog light, and high beam, which are represented by -1, 0, and 1 respectively; store the keyframe data with labels in Labeled database, the data in the labeled database is original video data and its labeled keyframes, expressed as (F _i,j ,k), where k is -1, 0 or 1.

In step (2), the construction method of the deep learning module based on CNN+LSE is to adopt the LeNet5 convolutional neural network structure. The module is divided into eight layers, the first six layers are the feature extraction part, and the last two layers are the classifier part. Among them, the feature extraction layer adopts the classic convolutional neural network structure, and the classifier layer adopts the fully connected structure; the module uses the data in the labeled database as the training data, and uses the CNN+LSE combination algorithm to train the deep learning module. For feature extraction The CNN method is used for training partly, while for the classifier layer, the LSE method is used for training, in order to realize the fast learning of module parameters and enhance the generalization ability of the module.

The specific method is as follows:

The key frame of the video in the label database is input to the first layer of the deep learning module based on CNN+LSE; in the second layer, different convolution kernels are used to perform convolution operations on the upper layer output; the third layer performs pooling on the upper layer output ( Downsampling) operation; the fourth layer and the fifth layer repeat the operation of the second layer and the third layer; the sixth layer expands the output features of the upper layer in turn and arranges them in a row; the seventh layer and the upper layer output features are fully interconnected; finally The first layer is also fully interconnected with the upper layer. The output of the deep learning module based on CNN+LSE is three situations: low beam, fog light and high beam, represented by -1, 0 and 1 respectively.

The training process of the deep learning module based on CNN+LSE is as follows:

Randomly take a sample (F _i,j ,k) from the labeled database, perform grayscale operation on F _i,j first to make it a grayscale image, and then convert the grayscaled key frame F _{i, j} ' is input into the module, that is, the input data is (F _i,j ',k); the two parts of the deep learning module are trained by CNN and LSE respectively; among them, the parameter training method of the feature extraction part is as follows:

(2-A1) Initialize all connection weight parameters of the feature extraction part in the deep learning module;

(2-A2) Calculate the actual output label O _k corresponding to the input key frame;

(2-A3) Calculate the difference between the actual output label O _k and the corresponding ideal output label k;

(2-A4) Weight learning: adjust the connection weight parameter matrix of the feature extraction part in the deep learning module through backpropagation by minimizing the error;

(2-A5) Until the key frames of all video data are traversed, the parameter training is completed;

The parameter training method of the classifier part is as follows:

(2-B1) The connection weight and bias between the rasterization layer and the fully connected layer are randomly generated, and the output of the fully connected layer is written

for the matrix

Where G( ) is the activation function, a _i is the connection weight, b _i is the bias, L is the number of fully connected layer nodes, N is the number of all key frames, x _j is the key frame, i=1,2 ,...,L,j=1,2,...,N;

(2-B2) Write the network output result of the corresponding key frame as an output vector Y=[y ₁ y ₂ ... y _n ] ^T , where y _j is the output label corresponding to the jth key frame x _j ;

(2-B3) Calculate the output weight β=PHY between the fully connected layer and the output layer, where P=(H ^T H) ⁻¹ .

In step (3), the data in the database of violation results to be checked is the video data judged to be violations by the recognition result processing module, and the results of violations to be checked should be manually checked, and then the confirmed information will be imported into the violation database, and Misjudged information will be deleted.

In step (3), the method for judging whether there is a high beam violation is: the key frame labeled as high beam with the next keyframe The time interval between ΔT=j ₂ -j ₁ , if ΔT≥θ, the vehicle has illegal use of high beams, where θ is the violation time threshold.

Beneficial effects of the present invention:

The invention automatically analyzes and recognizes the monitoring video, ensures the completeness of law enforcement evidence, is similar to manual judgment, has intelligence, and needs simple layout equipment, and can make full use of the original monitoring equipment. details as follows:

(1) Through the mining of video data, on the basis of ensuring accuracy, the adequacy of law enforcement evidence has been greatly improved, preventing the lack of evidence chains when high beams are illegally enforced;

(2) The same point requires less equipment, and a large number of monitoring equipment originally deployed can be directly reused, reducing costs and improving equipment utilization;

(3) The intelligent judgment of high beam violations based on video deep learning is used to replace manual law enforcement, realizing real automation and improving efficiency; at the same time, after deep learning, the recognition effect of high beam violations is expected to reach or Beyond the level of artificial recognition, it realizes the real intelligence of the recognition system;

(4) The deep learning module uses the method of CNN+LSE to learn the parameters of the system, which not only makes the parameter learning speed of the system faster, but also makes the generalization ability of the module stronger and the robustness of the system improved.

Description of drawings

Fig. 1 is a schematic diagram of the system structure of the present invention;

Figure 2 is a structure diagram of a deep learning module based on CNN+LSE.

detailed description

The present invention will be further described below in conjunction with the accompanying drawings and embodiments. It should be noted that the following description is only for explaining the present invention and not limiting its content.

As shown in Figure 1, a car high beam recognition system based on video deep learning includes the following two parts:

The foreground part is used to realize the identification and processing of high beam violations, including the road monitoring equipment module, video processing and identification module, recognition result processing module, and pending inspection results database;

The background part is used for video processing and deep learning of video, including key frame extraction algorithm, tagged database and deep learning module. The data in the label database is used for the training of the deep learning module, and the trained deep learning module and the key frame extraction algorithm are called by the video processing and recognition module.

Among them, the key frame extraction algorithm is a key frame extraction algorithm based on clustering; the deep learning module is a deep learning module based on CNN+LSE.

A car high beam recognition method based on video deep learning corresponding to the above system, the specific steps are as follows:

(1) The road monitoring equipment module obtains the driving video data of the car and transmits it to the video processing and recognition module.

(2) The video processing and recognition module calls the key frame extraction algorithm to extract the key frames of the original video data, and then performs the grayscale operation. The grayscaled key frames are used as input, and the trained CNN+LSE based on the label database is called The deep learning module obtains the output labels of each key frame, including low beam, fog light or high beam, and assigns the label to the corresponding key frame image.

The key frame extraction algorithm is as follows:

(2-1) Take the i-th segment V _i in the original video database, extract n frames at equal time intervals, and use F _i,j to name the frame at the j-th moment of the i-th segment video data, and set the key of the corresponding video data The frame sequence is expressed as {F _i,1 ,F _i,2 ,...,F _i,n }, where F _i,1 is the first frame and F _i,n is the last frame; define the similarity between two adjacent frames The degree is the similarity of the histograms of two adjacent frames (that is, the histogram feature difference), and the predefined threshold δ controls the density of clustering; wherein, i, j and n are all integers;

(2-2) Select the first frame F _i,1 as the initial clustering center, and calculate the similarity between the frame F _i,j and the initial clustering center, if the value is less than δ, it is determined that the frame and the clustering center The distance between the center frames is too large, therefore, F _{i, j} cannot be added to the cluster; if the similarity between F _{i, j} and all cluster centers is less than δ, then F _{i, j} forms a new cluster, F _{i, j} is the new cluster center; otherwise, add the frame to the cluster with the largest similarity, so that the distance between the frame and the center of this cluster is the smallest;

(2-3) Repeat the operation of (2-2) to classify the n frames extracted from the original video data V _i into different clusters, then you can select the key frame: extract the distance from each cluster The nearest frame of the cluster center is used as the representative frame of this cluster, and the representative frames of all clusters constitute the key frame of the original video data V _i .

The construction method of the labeled database is as follows:

A large amount of vehicle driving video data under the background of big data is used as the original video data, and the key frame extraction algorithm based on clustering is called on the original video data to extract the key frames, and the lighting type of the vehicle in the key frames is manually determined, and each key frame is added The label changes the original keyframe into labeled data. The label categories include: low beam, fog light, and high beam, which are represented by -1, 0, and 1 respectively; store the keyframe data with labels in Labeled database, the data in the labeled database is original video data and its labeled keyframes, expressed as (F _i,j ,k), where k is -1, 0 or 1.

As shown in Figure 2, the construction method of the deep learning module based on CNN+LSE is to adopt the LeNet5 convolutional neural network structure. The module is divided into eight layers. The first six layers are the feature extraction part, and the last two layers are the classifier part. Among them, the feature extraction layer adopts the classic convolutional neural network structure, and the classifier layer adopts the fully connected structure; the module uses the data in the labeled database as the training data, and uses the CNN+LSE combination algorithm to train the deep learning module. For feature extraction The CNN method is used for training partly, while for the classifier layer, the LSE method is used for training, in order to realize the fast learning of module parameters and enhance the generalization ability of the module. The specific method is as follows: the video key frames in the tagged database are input to the first layer of the deep learning module based on CNN+LSE; in the second layer, different convolution kernels are used to perform convolution operations on the upper layer output; Perform pooling (down-sampling) operations; the fourth and fifth layers repeat the operations of the second and third layers; the sixth layer expands the output features of the upper layer in turn and arranges them in a row; the seventh layer and the upper layer output features are all Interconnection; the last layer is also fully interconnected with the upper layer. The output of the deep learning module based on CNN+LSE is three situations: low beam, fog light and high beam, represented by -1, 0 and 1 respectively.

The training process of the deep learning module based on CNN+LSE is as follows:

Randomly take a sample (F _i,j ,k) from the labeled database, perform grayscale operation on F _i,j first to make it a grayscale image, and then convert the grayscaled key frame F _{i, j} ' is input into the module, that is, the input data is (F _i,j ',k); the two parts of the deep learning module are trained by CNN and LSE respectively; among them, the parameter training method of the feature extraction part is as follows:

(2-A1) Initialize all connection weight parameters of the feature extraction part in the deep learning module;

(2-A2) Calculate the actual output label O _k corresponding to the input key frame;

(2-A3) Calculate the difference between the actual output label O _k and the corresponding ideal output label k;

(2-A4) Weight learning: adjust the connection weight parameter matrix of the feature extraction part in the deep learning module through backpropagation by minimizing the error;

(2-A5) Until the key frames of all video data are traversed, the parameter training is completed;

The parameter training method of the classifier part is as follows:

(2-B1) The connection weight and bias between the rasterization layer and the fully connected layer are randomly generated, and the output of the fully connected layer is written

for the matrix

Where G( ) is the activation function, a _i is the connection weight, b _i is the bias, L is the number of fully connected layer nodes, N is the number of all key frames, x _j is the key frame, i=1,2 ,...,L,j=1,2,...,N;

(2-B2) Write the network output result of the corresponding key frame as an output vector Y=[y ₁ y ₂ ... y _n ] ^T , where y _j is the output label corresponding to the jth key frame x _j ;

(2-B3) Calculate the output weight β=PHY between the fully connected layer and the output layer, where P=(H ^T H) ⁻¹ .

(3) The original video data and the key frame with the label obtained in step (2) are used as the input of the recognition result processing module to judge whether the vehicle violates regulations, and the license plate recognition system is embedded in the recognition result processing module. When the target When the vehicle has high beam violations, the license plate is extracted, the vehicle information is obtained, and the suspected violation video data is imported into the violation result database to be inspected.

The method for judging whether there is a high beam violation is: the key frame labeled as high beam with the next keyframe The time interval between ΔT=j ₂ -j ₁ , if ΔT≥θ, the vehicle has illegal use of high beams, where θ is the violation time threshold.

(4) The data in the database of violation results to be checked is the video data judged to be violations by the identification result processing module. The information is deleted.

Although the specific implementation of the present invention has been described above in conjunction with the accompanying drawings, it does not limit the protection scope of the present invention. On the basis of the technical solution of the present invention, those skilled in the art can make various Modifications or variations are still within the protection scope of the present invention.

Claims (10)

1. A car high beam recognition system based on video deep learning, is characterized in that, comprises the following two parts: The foreground part is used to realize the identification and processing of high beam violations, including the road monitoring equipment module, video processing and identification module, recognition result processing module, and pending inspection results database; The background part is used for video processing and deep learning of video, including key frame extraction algorithm, tagged database and deep learning module. The data in the label database is used for the training of the deep learning module, and is called by the video processing and recognition module together with the trained deep learning module. 2. The system according to claim 1, wherein the key frame extraction algorithm is a cluster-based key frame extraction algorithm. 3. The system according to claim 1, wherein the deep learning module is a deep learning module based on CNN+LSE. 4. A kind of automobile high beam recognition method based on video deep learning corresponding to the system according to any one of claims 1 to 3, characterized in that, the specific steps are as follows: (1) The road monitoring equipment module obtains the driving video data of the car and transmits it to the video processing and identification module; (2) The video processing and recognition module calls the key frame extraction algorithm to extract the key frames of the original video data, and then performs the grayscale operation. The grayscaled key frames are used as input, and the trained CNN+LSE based on the label database is called The deep learning module obtains the output label of each key frame, including low beam, fog light or high beam, and assigns the label to the corresponding key frame image; (3) The original video data and the key frame with the label obtained in step (2) are used as the input of the recognition result processing module to judge whether the vehicle violates regulations, and the license plate recognition system is embedded in the recognition result processing module. When the target When the vehicle has high beam violations, the license plate is extracted, the vehicle information is obtained, and the suspected violation video data is imported into the violation result database to be inspected. 5. method according to claim 4, is characterized in that, in step (2), key frame extraction algorithm is as follows: (2-1) Take the i-th segment V _i in the original video database, extract n frames at equal time intervals, and use F _i,j to name the frame at the j-th moment of the i-th segment video data, and set the key of the corresponding video data The frame sequence is expressed as {F _i,1 ,F _i,2 ,...,F _i,n }, where F _i , ₁ is the first frame, and F _i,n is the last frame; define the similarity between two adjacent frames The degree is the similarity of the histograms of two adjacent frames (that is, the histogram feature difference), and the predefined threshold δ controls the density of clustering; wherein, i, j and n are all integers; (2-2) Select the first frame F _i,1 as the initial clustering center, and calculate the similarity between the frame F _i,j and the initial clustering center, if the value is less than δ, it is determined that the frame and the clustering center The distance between the center frames is too large, therefore, F _{i, j} cannot be added to the cluster; if the similarity between F _{i, j} and all cluster centers is less than δ, then F _{i, j} forms a new cluster, F _{i, j} is the new cluster center; otherwise, add the frame to the cluster with the largest similarity, so that the distance between the frame and the center of this cluster is the smallest; (2-3) Repeat the operation of (2-2) to classify the n frames extracted from the original video data V _i into different clusters, then you can select the key frame: extract the distance from each cluster The nearest frame of the cluster center is used as the representative frame of this cluster, and the representative frames of all clusters constitute the key frame of the original video data V _i . 6. method according to claim 5, is characterized in that, in step (2), the construction method of band label database is as follows: A large amount of vehicle driving video data under the background of big data is used as the original video data, and the key frame extraction algorithm based on clustering is called on the original video data to extract the key frames, and the lighting type of the vehicle in the key frames is manually determined, and each key frame is added The label turns the original keyframe into labeled data. The label categories include: low beam, fog light, and high beam, which are represented by -1, 0, and 1 respectively; store the keyframe data with labels in Labeled database, the data in the labeled database is original video data and its labeled keyframes, expressed as (F _i,j ,k), where k is -1, 0 or 1. 7. method according to claim 6, it is characterized in that, in step (2), the construction method based on the deep learning module of CNN+LSE is, adopts LeNet5 convolutional neural network structure, module is divided into eight layers altogether, preceding The six layers are the feature extraction part, and the last two layers are the classifier part. Among them, the feature extraction layer adopts the classic convolutional neural network structure, and the classifier layer adopts the fully connected structure; the data in the labeled database is used as the training data, using CNN The +LSE combination algorithm trains the deep learning module, uses the CNN method for the feature extraction part, and uses the LSE method for the classifier layer. 8. method according to claim 7, is characterized in that, the training process based on the deep learning module of CNN+LSE is as follows: Randomly take a sample (F _i,j ,k) from the labeled database, perform grayscale operation on F _i,j first to make it a grayscale image, and then convert the grayscaled key frame F _{i, j} ' is input into the module, that is, the input data is (F _i,j ',k); the two parts of the deep learning module are trained by CNN and LSE respectively; among them, the parameter training method of the feature extraction part is as follows: (2-A1) Initialize all connection weight parameters of the feature extraction part in the deep learning module; (2-A2) Calculate the actual output label O _k corresponding to the input key frame; (2-A3) Calculate the difference between the actual output label O _k and the corresponding ideal output label k; (2-A4) Weight learning: adjust the connection weight parameter matrix of the feature extraction part in the deep learning module through backpropagation by minimizing the error; (2-A5) Until the key frames of all video data are traversed, the parameter training is completed; The parameter training method of the classifier part is as follows: (2-B1) The connection weight and bias between the rasterization layer and the fully connected layer are randomly generated, and the output of the fully connected layer is written as a matrix Where G( ) is the activation function, a _i is the connection weight, b _i is the bias, L is the number of fully connected layer nodes, N is the number of all key frames, x _j is the key frame, i=1,2 ,...,L,j=1,2,...,N; (2-B2) Write the network output result of the corresponding key frame as an output vector Y=[y ₁ y ₂ ... y _n ] ^T , where y _j is the output label corresponding to the jth key frame x _j ; (2-B3) Calculate the output weight β=PHY between the fully connected layer and the output layer, where P=(H ^T H) ⁻¹ . 9. method according to claim 4, it is characterized in that, in step (3), the data in the results database of violations to be checked is the video data that is judged as violations by the identification result processing module, and the results of violations to be checked wherein should accept Manual inspection, and then import the confirmed correct information into the violation database, and delete the misjudged information. 10. The method according to claim 8, characterized in that, in step (3), the judging method for whether there is a high beam violation is: the label is the key frame of the high beam with the next keyframe The time interval between ΔT=j ₂ -j ₁ , if ΔT≥θ, the vehicle has illegal use of high beams, where θ is the violation time threshold.