CN111583130A - Method for recovering license plate image for LPR - Google Patents

Abstract

The invention discloses a method for recovering a license plate image for LPR. The recognized image restores the model of the network, trains it with the training set, and obtains the corresponding training model; uses the validation set to test the accuracy of the training model, and then adjusts the hyperparameters of the model and optimizes the model to obtain better performance; Set the image input to the determined optimal model, test its generalization performance, and observe how the recovery effect of the license plate image is. In this scheme, the structure of the license plate image restoration network is redesigned, an auxiliary network is added to optimize the restoration quality of the image, and the robustness of the LPR is significantly increased. Therefore, the accuracy of license plate recognition is also quite high, and it is a fast and accurate recognition network.

Description

A method for recovering license plate images for LPR

technical field

The invention relates to the technical field of image processing, and in particular, to a method for recovering a license plate image for LPR.

Background technique

With the continuous improvement of the country's economic strength, people's living standards have also been greatly improved, and more and more families have their own private cars. The accompanying traffic problems are also increasing day by day. Today, the problem of traffic vehicle management is one of the important problems in urban management. To this end, the Intelligent Transportation System (hereinafter referred to as ITS) came into being.

ITS is a combination of existing science and technology (such as computer counting, sensor technology, image processing technology, etc.) for transportation, service control, etc., and license plate recognition (License Plate Recognition, hereinafter referred to as LPR) is one of the important basic link. At present, the commonly used method is to use the color and texture features of the license plate to locate the license plate area, and then perform appropriate enhancement processing on the image, and then recognize the image to obtain the license plate information.

Due to the development and application of Convolutional Neural Networks (CNN for short), many tasks in the field of computer vision have been greatly developed, and many LPR methods based on CNN have also been applied to solve the problem of recognizing license plate images in the real world. However, many existing methods are based on captured high-quality images. As for low-quality images captured in harsh environments (such as strong light, night, haze, etc.), or captured in motion, it is still difficult to identify.

SUMMARY OF THE INVENTION

The purpose of the present invention is to overcome the deficiencies of the prior art, and to provide a method for recovering a license plate image with strong robustness, high recognition accuracy and good performance.

The object of the present invention is achieved through the following technical solutions:

A method for recovering a license plate image for LPR, the method mainly includes the following specific steps:

Step S1: After a series of operations are performed on the images in the known data set, the images are divided into training sets, validation sets and test sets in proportion.

Specifically, the step S1 further includes the following steps:

Step S11: Using several well-known license plate recognition data sets VTLP, the training set, the verification set and the test set are divided according to the ratio of 6:2:2.

i∈{-30°,-15°,+15°,+30°}，尺寸变换后的子图记为

采取逐像素的二值分割后的图片记为

字符计数值为C。Step S12: In order to increase the amount of training data, the training set is then rotated with different angles to generate four sub-images, which are doubled by size transformation and segmentation methods; the original training image is denoted as I ^H , and divided into four sub-images. The rotated subgraph is denoted as

i∈{-30°,-15°,+15°,+30°}, the size-transformed subgraph is denoted as

The image after pixel-by-pixel binary segmentation is denoted as

The character count value is C.

As a preferred solution of the present invention, in the step S1, a series of operations include averaging, dehazing, and cropping operations.

Step S2: Establish a model of an image restoration network for license plate recognition, train it with a training set, and obtain a corresponding training model.

Specifically, the step S2 further includes the following steps:

S21: Set the backbone network and auxiliary network of the restoration network, the backbone network includes two sub-networks, the auxiliary network includes two decoder modules, and then train the sub-networks and modules respectively;

S22: Train the noise reduction sub-network _MD ;

S23: training the correction sub-network _MR ;

S24: Train the pixel segmentation module A _S ;

S25: training character technology module A _C ;

S26: Add the weights of the four loss functions.

Step S3: Use the validation set to test the accuracy of the training model, then adjust the hyperparameters of the model, and optimize the model to obtain better performance.

进而调节模型的超参数。Further, the step S3 further includes: after the training model is obtained through step S2, the LPR network can obtain the output image result provided by the correction module through the connection with the correction sub-network

Then adjust the hyperparameters of the model.

Step S4: Input the test set image into the determined optimal model, test its generalization performance, and observe how the recovery effect of the license plate image is.

Further, the step S4 further includes: providing the test set pictures to the license plate recognition network, and obtaining the recognition result LPR _result through the license plate image restoration network and the LPR network respectively.

The working process and principle of the present invention are as follows: a method for recovering a license plate image for LPR provided by the present invention redesigns the structure of the license plate image recovery network, adds an auxiliary network to optimize the image recovery quality, and makes the LPR robust The robustness is significantly increased; in addition, the restoration network provided by this scheme adopts the method of combining denoising and correction network, and obtains good results, and the LPR network adopts the current detector with high accuracy and fast recognition, so the license plate recognition The accuracy rate is also quite high, and it is a fast and accurate recognition network.

Compared with the prior art, the present invention also has the following advantages:

(1) The method for recovering license plate images for LPR provided by the present invention can handle and restore low-quality license plate images well, and the recovered images can be accurately recognized by the currently popular LPR network, which is suitable for different environments. Capture the obtained license plate image.

(2) The method for recovering license plate images for LPR provided by the present invention is aimed at low-quality images, and recovers images through a series of end-to-end convolutional neural networks based on deep learning, and then takes the recovered images to LPR. Identifying on the network can significantly reduce interference factors and quickly and accurately identify license plates.

(3) The method for recovering a license plate image for LPR provided by the present invention is designed for the structure of a license plate image recovery network, an auxiliary network is added to optimize the image recovery quality, and the robustness of the license plate recognition can be enhanced. The restoration network of the present invention adopts the combination of denoising and correction network, and adds an auxiliary network structure to maximize the quality of the restored image, which is unique and innovative.

Description of drawings

FIG. 1 is a flowchart of a method for recovering a license plate image for LPR provided by the present invention.

FIG. 2 is a schematic structural diagram of the entire license plate recognition network provided by the present invention.

FIG. 3 is a schematic structural diagram of a U-Net-based network provided by the present invention.

Detailed ways

In order to make the objectives, technical solutions and advantages of the present invention clearer and clearer, the present invention will be further described below with reference to the accompanying drawings and examples.

Example 1:

As shown in FIG. 1 to FIG. 3 , the present embodiment discloses a method for recovering a license plate image for LPR, and the method mainly includes the following specific steps:

Step S1: After a series of operations are performed on the images in the known data set, the images are divided into training sets, validation sets and test sets in proportion.

Specifically, the step S1 further includes the following steps:

Step S11: Using several well-known license plate recognition data sets VTLP, the training set, the verification set and the test set are divided according to the ratio of 6:2:2.

i∈{-30°,-15°,+15°,+30°}，尺寸变换后的子图记为

采取逐像素的二值分割后的图片记为

字符计数值为C。Step S12: In order to increase the amount of training data, the training set is then rotated with different angles to generate four sub-images, which are doubled by size transformation and segmentation methods; the original training image is denoted as I ^H , and divided into four sub-images. The rotated subgraph is denoted as

i∈{-30°,-15°,+15°,+30°}, the size-transformed subgraph is denoted as

The image after pixel-by-pixel binary segmentation is denoted as

The character count value is C.

As a preferred solution of the present invention, in the step S1, a series of operations include averaging, dehazing, and cropping operations.

Step S2: Establish a model of an image restoration network for license plate recognition, train it with a training set, and obtain a corresponding training model.

Specifically, the step S2 further includes the following steps:

S21: Set the backbone network and auxiliary network of the restoration network, the backbone network includes two sub-networks, the auxiliary network includes two decoder modules, and then train the sub-networks and modules respectively;

S22: Train the noise reduction sub-network _MD ;

S23: training the correction sub-network _MR ;

S24: Train the pixel segmentation module A _S ;

S25: training character technology module A _C ;

S26: Add the weights of the four loss functions.

Step S3: Use the validation set to test the accuracy of the training model, then adjust the hyperparameters of the model, and optimize the model to obtain better performance.

进而调节模型的超参数。Further, the step S3 further includes: after the training model is obtained through step S2, the LPR network can obtain the output image result provided by the correction module through the connection with the correction sub-network

Then adjust the hyperparameters of the model.

Step S4: Input the test set image into the determined optimal model, test its generalization performance, and observe how the recovery effect of the license plate image is.

Further, the step S4 further includes: providing the test set pictures to the license plate recognition network, and obtaining the recognition result LPR _result through the license plate image restoration network and the LPR network respectively.

The working process and principle of the present invention are as follows: a method for recovering a license plate image for LPR provided by the present invention redesigns the structure of the license plate image recovery network, adds an auxiliary network to optimize the image recovery quality, and makes the LPR robust The robustness is significantly increased; in addition, the restoration network provided by this scheme adopts the method of combining denoising and correction network, and obtains good results, and the LPR network adopts the current high-accuracy and fast-recognizing detector, so license plate recognition The accuracy rate is also quite high, and it is a fast and accurate recognition network.

Example 2:

Referring to FIG. 1 to FIG. 3 , the present embodiment discloses a method for recovering a license plate image for LPR, including the following steps:

S1: After performing a series of operations such as averaging, dehazing, and cropping on the images in the known data set, the size of the generated image is 572*572, and the image is divided into training set, validation set and test set proportionally.

The specific steps include the following:

S11: The data set VTLP is used, which contains 10650 license plate images, and then it is divided into training set, validation set and test set according to the ratio of 6:2:2, containing 6390, 2130 and 2130 license plate images respectively.

i∈{-30°,-15°,+15°,+30°}，进行尺寸变换后生成四张子

二值分割后的子图记为

把字符标签记为C；S12: Then expand the training set with the correct label, and a label image I ^H is expanded into four sub-images through rotation transformation

i∈{-30°,-15°,+15°,+30°}, after the size transformation, four sheets are generated

The subgraph after binary segmentation is denoted as

Let the character label be C;

S2: Establish an image restoration network model for license plate recognition, train it with a training set, and obtain a corresponding training model. The specific steps include the following:

S21: The restoration network includes a backbone network and an auxiliary network, wherein the backbone network includes two sub-networks, and the first sub-network takes a low-quality image as input and outputs a restored image. The second is the correction network, which corrects the output from the noise reduction network. The auxiliary network includes a text counting module and a pixel segmentation module. Their input is the sum of the encoder image features of the noise reduction network and the correction network, and the output is the number of characters and the image after binary segmentation, respectively. The restoration network of the present invention adopts the U-Net structure, the noise reduction network and the correction network both include encoder and decoder modules, while the counting network and the segmentation network have only the decoder module. Both sub-networks are built based on the U-Net structure, as shown in Figure 3, which is the U-Net structure. The reason for choosing this structure is that the U-Net structure can improve the details of the target in the image. Image generation has a negative impact. We adopt a U-Net-based structure while adding skip connections, which can share low-level semantic information of images.

S211: Both sub-networks of the backbone network include an encoder and a decoder, wherein the network structure of the encoder includes a first convolutional layer conv1(1,2)(3×3×32)×2, with a stride of 2; the second Convolutional layer conv2(1,2)(3×3×64)×2, stride 2; third convolutional layer conv3(1,2)(3×3×128)×2, stride 2; fourth Convolutional layer conv4(1,2)(3×3×256)×2, stride 2; fifth convolutional layer conv5(1,2)(3×3×512)×2, stride 2; Pooling layer pool1(2×2)max pooling, stride 2; second pooling layer pool2(2×2)max pooling, stride 2; third pooling layer pool3(2×2)max pooling, stride 2; The fourth pooling layer pool4 (2×2) max pooling, step size 2; it should be noted that a BN module is added before each convolutional layer, and the activation function connected after the convolutional layer uses LeakyReLU function, the feature map generated after the second convolution kernel in each convolutional layer will be extracted to the decoder network through skip connections. There is a corresponding max pooling layer between the layers.

S212: The network structures of the two decoders adopted by the sub-network of the backbone network and the decoder adopted by the pixel segmentation module of the auxiliary network are the same. The decoder includes: sixth convolutional layer conv6(1,2)(3×3×256)×2, stride 2; seventh convolutional layer conv7(1,2)(3×3×128)×2 , stride 2; the eighth convolutional layer conv8(1,2)(3×3×64)×2, stride 2; the ninth convolutional layer conv9(1,2)(3×3×32)×2 , stride 2; tenth convolution layer conv10 (1×1×3), stride 1; first upsampling layer upsample: concatenate con5_2 and conv4_2; second upsampling layer upsample: concatenate con6_2 and conv3_2 ; The third upsampling layer upsample: concatenate con7_2 and conv2_2; the fourth upsampling layer upsample: concatenate con8_2 and conv1_2. The upsampling layer is connected between two convolutional layers, and each convolutional layer is passed through the LeakyReLU activation function.

S213: The text counting module of the auxiliary network uses CNN to predict the number of texts to complete a simple classification task, wherein the decoder network structure includes five convolutional layers, and the dimensions of the five convolutional layers are (N×N×512 ), (1×1×256), (1×1×128), (1×1×64), (1×1×1).

S214: The connection mode of the backbone network is as follows: the low-quality picture input to the restoration network first passes through the noise reduction sub-network, and then the output result is input to the correction sub-network. In this, the encoder of the noise reduction network and the correction network will lead to an output respectively. The sum of the output feature maps of the two encoders is recorded as a feature set F and then fed to the counting decoder and segmentation in the auxiliary network respectively. decoder. The text count can accurately separate each text character of the license plate, predict the number of text, and make the image more suitable for the subsequent LPR network for detection. Pixel segmentation can make the picture of the license plate clearer and easier for text recognition.

i∈{-30°,-15°,+15°,+30°}喂给降噪子网络M_D用来训练，其中w是降噪网络的参数，此子网络的损失函数表示如下：S22: Subgraph

i∈{-30°,-15°,+15°,+30°} is fed to the noise reduction sub-network _MD for training, where w is the parameter of the noise reduction network, and the loss function of this sub-network is expressed as follows:

喂给校正子网络M_R用来训练，其中w是校正子网络的参数，其损失函数表达如下：S23: Subgraph

It is fed to the syndrome network _MR for training, where w is the parameter of the syndrome network, and its loss function is expressed as follows:

喂给像素分割解码器模块A_S用以训练，其中F表示主网络的两个子网络的编码器求和之后的特征集合，

表明该某个像素的是车牌区域的概率。

表明像素真实分类，交叉熵损失函数公式如下：S24: Subgraph

is fed to the pixel segmentation decoder module A _S for training, where F represents the feature set after the summation of the encoders of the two sub-networks of the main network,

Indicates the probability that a certain pixel is a license plate area.

Indicates the true classification of pixels, and the formula of the cross-entropy loss function is as follows:

S25: The count label value C is fed to the pixel segmentation decoder module A _C for training, C _pred represents the predicted value, C _{G, T} represent the label value, and the loss function is as follows:

S26: Finally, the loss function of the entire restoration network can be expressed as the weight sum of all sub-objective functions:

S27: According to this total loss function, parameters can be adjusted to train the optimal model

S3: After each epoch is completed, use the validation set to test the accuracy of the training model, and then adjust the hyperparameters of the model and optimize the model to obtain better performance. The specific steps are as follows:

然后输入到LPR网络中进行目标检测，可以得到图片的准确率结果，然后根据结果可以适当来调节超参数优化模型。S31: According to the training model obtained in S2, input the license plate picture of the verification set into the recovery network, and obtain the output result

Then input it into the LPR network for target detection, and the accuracy results of the pictures can be obtained, and then the hyperparameter optimization model can be appropriately adjusted according to the results.

S4: Input the test set image into the determined optimal model, test its generalization performance, and observe the recovery effect of the license plate image. The specific steps include:

S41: Input the image I _test of the test set to the restoration network again, and input the output result to the LPR network, and the obtained output can be expressed as:

LPR _result = LPR(MR ( _{M D} (I _test )))

S42: Then compare the accuracy of the judgment result with the existing SOTA license plate recognition network model. It is concluded that the network performance of the present invention is excellent, and it is the optimal network in the network structure using the same data set at present, and the recognition accuracy rate is as high as 93%.

The above-mentioned embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited by the above-mentioned embodiments, and any other changes, modifications, substitutions, combinations, The simplification should be equivalent replacement manners, which are all included in the protection scope of the present invention.

Claims (6)

1. A method for recovering a license plate image for LPR is characterized by comprising the following steps:

step S1: after a series of operations are carried out on the images in the known data set, the images are proportionally divided into a training set, a verification set and a test set;

step S2: establishing a model of an image recovery network for license plate recognition, and training the model by using a training set to obtain a corresponding training model;

step S3: the verification set is used for checking the accuracy of the training model, so that the hyper-parameters of the model are adjusted, and the model is optimized to obtain better performance;

step S4: and inputting the test set image into the determined optimal model, testing the generalization performance of the test set image, and observing the recovery effect of the license plate image.

2. The method for restoring a license plate image for LPR of claim 1, wherein said step of S1 further comprises the steps of:

step S11: adopting a plurality of famous license plate recognition data sets VTLP, and carrying out verification and test on a training set, a verification set and a test set according to the following steps of 6:2:2 to divide;

step S12: to increase the amount of training data, the pairsThe training set generates four sub-pictures by adopting rotation with different angles, and doubles the four sub-pictures by a size conversion and segmentation method; the original training picture is marked as I^HThe divided four rotated subgraphs are

The subgraph after the size transformation is marked as

The picture after binary segmentation by pixel is recorded as

The character count value is C.

3. The method for restoring a license plate image for LPR of claim 1, wherein said step of S2 further comprises the steps of:

s21: setting a main network and an auxiliary network of a recovery network, wherein the main network comprises two sub-networks, and the auxiliary network comprises two decoder modules and then respectively trains the sub-networks and the modules;

s22: training noise reduction subnetwork M_D；

S23: training the syndrome network M_R；

S24: training pixel segmentation module A_S；

S25: training character technology module A_C；

S26: the four loss function weights are added.

4. The method for restoring a license plate image for LPR of claim 1, wherein said step of S3 further comprises: after obtaining the training model through step S2, the LPR network can obtain the output image result provided by the correction module through connection with the syndrome network

And then adjust the modelAnd (4) super-parameter.

5. The method for restoring a license plate image for LPR of claim 1, wherein said step of S4 further comprises: providing the test set picture to a license plate recognition network, and respectively obtaining recognition results LPR through a license plate image recovery network and an LPR network_result。

6. The method for restoring a license plate image for LPR of claim 1, wherein said sequence of operations of step S1 comprises averaging, defogging and cropping.

Images