Disclosure of Invention
In order to overcome the defects of the prior art, one of the purposes of the invention is to provide a highway sign identification method, which adopts an automatic image identification mode to automatically judge the traffic sign of the highway, and improves the working efficiency and the identification precision of the traffic sign identification of vehicles in the running process of the highway.
The invention provides a highway sign identification method, which comprises the following steps:
acquiring an image to be identified, and acquiring an RGB image and a depth image of a highway traffic sign;
the method comprises the steps of multi-scale zooming images, mapping depth information of the depth images to the RGB images to generate fusion images, and carrying out multi-scale zooming on the fusion images to generate a plurality of zooming images;
generating a characteristic image, calculating the characteristic image containing color information and gradient information of the zoomed image, calculating an integral image of the characteristic image, and generating a selection characteristic image by taking the characteristics of the integral image as the input of a characteristic selection classifier;
and sliding the traversal image, traversing the selected characteristic image through a sliding window, and identifying the traffic sign in the sliding window.
Further, the method also comprises the following steps of extracting a traffic sign area: dividing the selected characteristic image into a plurality of sub-regions along the width direction, traversing each sub-region, calculating the gradient amplitude and the gradient angle of each sub-region, calculating the gradient amplitude sum of the gradient angle in each sub-region from +/-1 DEG to +/-6 DEG, and generating a traffic sign region in the selected characteristic image.
Further, the step of extracting the traffic sign region further comprises: and performing secondary area extraction on the traffic sign area by adopting the depth value, acquiring an area with a fixed value of the depth value change rate in the length direction, and generating an optimized traffic sign area.
Further, the step of generating the feature image further includes estimating gradient integral features of the scaled image of different scales by using an interpolation algorithm, generating the integral image according to the gradient integral features, and selecting the features of the integral image by using a feature selection classifier with an increasing number of layers to generate a selected feature image.
And further, detecting context information, namely inputting the spatial context information and the temporal context information of the fused image into the feature selection classifier, and selecting the traffic sign with the highest feature score in the overlapped boxes.
An electronic device, comprising: a processor;
a memory; and a program, wherein the program is stored in the memory and configured to be executed by the processor, the program comprising instructions for performing the highway sign recognition method described above.
A computer-readable storage medium, on which a computer program is stored, which computer program is executed by a processor to carry out the above-mentioned highway sign recognition method.
A highway sign identification system comprising:
the module for acquiring the image to be identified comprises: the system comprises a display device, a display device and a display device, wherein the display device is used for displaying an image of the traffic sign of the expressway;
multi-scale scaling the image module: the depth information of the depth image is mapped to the RGB image to generate a fusion image, and the fusion image is subjected to multi-scale scaling to generate a plurality of scaling images;
a feature image generation module: the feature image which is used for calculating the zoom image and contains color information and gradient information, the integral image of the feature image is calculated, and the feature of the integral image is used as the input of a feature selection classifier to generate a selection feature image;
sliding and traversing the image module: and traversing the selected characteristic image through a sliding window, and identifying the traffic sign in the sliding window.
Further, the module for extracting the traffic sign area comprises the following modules: the system comprises a selection characteristic image, a gradient amplitude value and a gradient angle, a gradient amplitude value sum of the gradient angle in each sub-region from +/-1 degrees to +/-6 degrees, and a traffic sign region in the selection characteristic image, wherein the selection characteristic image is divided into a plurality of sub-regions along the width direction, each sub-region is traversed, the gradient amplitude value and the gradient angle of each sub-region are calculated, and the traffic sign region in the selection characteristic image is generated; the module for extracting the traffic sign area further comprises the step of performing secondary area extraction on the traffic sign area by adopting the depth value, acquiring an area with a fixed depth value change rate along the length direction, and generating an optimized traffic sign area.
Further, the feature image generation module further estimates gradient integral features of the scaled images of different scales by adopting an interpolation algorithm, generates the integral image according to the gradient integral features, and selects the features of the integral image by adopting a feature selection classifier with increasing layer number to generate a selected feature image; the system also comprises a context information detection module: and the system is used for inputting the spatial context information and the temporal context information of the fused image into the feature selection classifier and selecting the traffic sign with the highest feature score in the overlapped boxes.
Compared with the prior art, the invention has the beneficial effects that:
the invention provides a highway sign identification method, which comprises the following steps: acquiring an image to be identified, scaling the image in multiple scales, generating a characteristic image, traversing the image in a sliding manner, and acquiring an RGB (red, green and blue) image and a depth image of a highway traffic sign; mapping the depth information of the depth image to an RGB image to generate a fusion image, and carrying out multi-scale scaling on the fusion image to generate a plurality of scaled images; calculating a feature image containing color information and gradient information of the zoomed image, calculating an integral image of the feature image, and generating a selection feature image by taking the features of the integral image as the input of a feature selection classifier; and traversing the selected characteristic image through the sliding window, and identifying the traffic sign in the sliding window. The invention also relates to a storage medium, electronic equipment and a highway sign identification system, and the invention adopts an automatic image identification mode to automatically judge the highway traffic sign, thereby improving the working efficiency and the identification precision of the traffic sign identification of vehicles in the running process of the highway.
The foregoing description is only an overview of the technical solutions of the present invention, and in order to make the technical solutions of the present invention more clearly understood and to implement them in accordance with the contents of the description, the following detailed description is given with reference to the preferred embodiments of the present invention and the accompanying drawings. The detailed description of the present invention is given in detail by the following examples and the accompanying drawings.
Detailed Description
The present invention will be further described with reference to the accompanying drawings and the detailed description, and it should be noted that any combination of the embodiments or technical features described below can be used to form a new embodiment without conflict.
The method for identifying the highway sign, as shown in fig. 1, comprises the following steps:
acquiring an image to be identified, and acquiring an RGB image and a depth image of a highway traffic sign; in this embodiment, an RGB image acquired by an industrial camera is acquired, and a depth image acquired by a depth camera is acquired, where the RGB image and the depth image are images acquired by the same target at the same time. When the image is acquired, the industrial camera and the depth camera are in the same acquisition size, the image at the same position is acquired at the same time, in order to keep the images acquired by the two cameras to eliminate the visual angle difference as much as possible, the industrial camera and the depth camera are vertically arranged, and the lens direction is the horizontal direction.
And (3) multi-scale zooming the image, mapping the depth information of the depth image to the RGB image to generate a fusion image, and carrying out multi-scale zooming on the fusion image to generate a plurality of zoomed images.
Generating a characteristic image, calculating the characteristic image containing color information and gradient information of the zoomed image, calculating an integral image of the characteristic image, and generating a selection characteristic image by taking the characteristics of the integral image as the input of a characteristic selection classifier; in one embodiment, the original RGB image is color-converted into a LUV space with a low coupling degree, and 3 color feature images, 6 gradient feature images in different directions, and l gradient images including gradient size are obtained by calculation; calculating an integral image corresponding to each characteristic image; randomly selecting any one feature image, rectangular frames with any position and any size, and quickly calculating an integral value in the rectangular frames through an integrogram to obtain the l-dimensional features. And (3) performing feature selection by adopting an Adaboost algorithm, obtaining a final score through the weighted sum of a plurality of weak classifiers, generating a strong classifier, performing secondary classification, and realizing that detection is completed by only using a small amount of features in actual detection.
In one embodiment, the method further comprises the step of extracting the traffic sign area: dividing the selected characteristic image into a plurality of sub-regions along the width direction, traversing each sub-region, calculating the gradient amplitude and the gradient angle of each sub-region, calculating the gradient amplitude sum of the gradient angle in each sub-region from +/-1 degrees to +/-6 degrees, and generating the traffic sign region in the selected characteristic image. The step of extracting the traffic sign area further comprises: and performing secondary area extraction on the traffic sign area by adopting the depth value, acquiring the area with the depth value change rate in the length direction as a fixed value, and generating an optimized traffic sign area, wherein the depth value is check information, and because the depth value in the background changes into nonlinear change and the depth value in the length direction on the traffic sign continuously changes or keeps unchanged, performing second-order derivation on the depth value, and when the second-order derivation of the depth value change rate is zero, assisting in verifying the correctness of the traffic sign area extracted in the area.
In an embodiment, the information of the gradient does not have scale invariance, and the proportional relation of the gradient integral values in the same area cannot be directly estimated in images with different scaling sizes; the method comprises the following steps of amplifying an original image into an up-sampling image, reducing the original image into a down-sampling image, and estimating integral gradient information of the up-sampling image and the down-sampling image according to the following formula:
wherein, I is an original image; f (I, S) is the original image scaling k is 2SThen, estimating a formula by using the gradient integral characteristic; when S is>0,k>1 denotes an up-sampled image; when S is<0,k<1 denotes a down-sampled image.
In an embodiment, preferably, the step of generating the feature image further includes selecting features of the integral image by using a feature selection classifier with an increasing number of layers to generate the selected feature image. In order to process the traffic sign detection task with a large number of negative samples, a simple classifier is used for roughly filtering the samples to filter out samples which are obviously not marked; and then fine filtering is performed by a complex classifier. Specifically, an Adaboost classifier with the maximum number of layers being T is adopted, and the complexity and the classification capability of the classifier are increased through the Adaboost classifier with the number of layers increasing, so that a large number of negative samples are rejected in the early stage.
And sliding the traversal image, traversing the selected characteristic image through a sliding window, and detecting the traffic sign in the sliding window.
In one embodiment, the spatial context information includes location, size, width; the time context information is the image, the horizontal axis coordinate, the vertical axis coordinate and the width of a sliding window in the original image which needs to be detected by the current frame, preferably, the method also comprises the step of detecting the context information, inputting the space context information and the time context information of the fused image into a feature selection classifier, and selecting the traffic sign with the highest feature score in the overlapped boxes.
An electronic device, comprising: a processor; a memory; and a program, wherein the program is stored in the memory and configured to be executed by the processor, the program comprising instructions for performing the highway sign identification method described above; a computer-readable storage medium, on which a computer program is stored, the computer program being executed by a processor for the above-mentioned highway marking identification method.
The highway sign recognition system, as shown in fig. 2, includes:
the image to be recognized is acquired by the image acquiring module to acquire the RGB image and the depth image of the highway traffic sign.
The multi-scale image scaling module maps the depth information of the depth image to the RGB image to generate a fusion image, and performs multi-scale scaling on the fusion image to generate a plurality of scaling images.
The feature image generation module calculates a feature image containing color information and gradient information of the zoomed image, calculates an integral image of the feature image, and generates a selection feature image by taking the features of the integral image as the input of the feature selection classifier; in one embodiment, the feature image generation module performs color conversion on the original RGB image into an LUV space with a low coupling degree, and calculates to obtain 3 color feature images, 6 gradient feature images in different directions, and l gradient images including gradient sizes; calculating an integral image corresponding to each characteristic image; randomly selecting any one feature image, rectangular frames with any position and any size, and quickly calculating an integral value in the rectangular frames through an integrogram to obtain the l-dimensional features. And the characteristic image generation module adopts an Adaboost algorithm to perform characteristic selection, obtains a final score through the weighted sum of a plurality of weak classifiers, generates a strong classifier, performs secondary classification, and realizes that detection is completed by only using a small amount of characteristics in actual detection.
In one embodiment, the method further comprises the step of extracting the traffic sign area module: dividing the selected characteristic image into a plurality of sub-regions along the width direction, traversing each sub-region, calculating the gradient amplitude and the gradient angle of each sub-region, calculating the gradient amplitude sum of the gradient angle in each sub-region from +/-1 degrees to +/-6 degrees, and generating the traffic sign region in the selected characteristic image. The module for extracting the traffic sign area further comprises: and performing secondary area extraction on the traffic sign area by adopting the depth value, acquiring the area with the depth value change rate in the length direction as a fixed value, and generating an optimized traffic sign area, wherein the depth value is check information, and because the depth value in the background changes into nonlinear change and the depth value in the length direction on the traffic sign continuously changes or keeps unchanged, performing second-order derivation on the depth value, and when the second-order derivation of the depth value change rate is zero, assisting in verifying the correctness of the traffic sign area extracted in the area.
In an embodiment, the information of the gradient does not have scale invariance, and the proportional relation of the gradient integral values in the same area cannot be directly estimated in images with different scaling sizes; the method comprises the following steps of amplifying an original image into an up-sampling image, reducing the original image into a down-sampling image, and estimating integral gradient information of the up-sampling image and the down-sampling image according to the following formula:
wherein, I is an original image; f (I, S) is the original image scaling k is 2SThen, estimating a formula by using the gradient integral characteristic; when S is>0,k>1 denotes an up-sampled image; when S is<0,k<1 denotes a down-sampled image.
In an embodiment, preferably, the feature image generation module selects features of the integral image by using a feature selection classifier with an increasing number of layers to generate the selected feature image. In order to process the traffic sign detection task with a large number of negative samples, a characteristic image module is selected to firstly carry out coarse filtration on the samples by using a simple classifier, and the samples which are obviously not signs are filtered; and then fine filtering is performed by a complex classifier. Specifically, an Adaboost classifier with the maximum number of layers being T is adopted, and the complexity and the classification capability of the classifier are increased through the Adaboost classifier with the number of layers increasing, so that a large number of negative samples are rejected in the early stage.
And the sliding traversal image module traverses the selected characteristic image through the sliding window and detects the traffic sign in the sliding window.
In one embodiment, the spatial context information includes location, size, width; the time context information is the image, the horizontal axis coordinate, the vertical axis coordinate and the width of a sliding window in the original image to be detected of the current frame, preferably, the method further comprises the step that a context information detection module inputs the space context information and the time context information of the fused image into a feature selection classifier, and selects the traffic sign with the highest feature score in the overlapped boxes.
The invention provides a highway sign identification method, which comprises the following steps: acquiring an image to be identified, scaling the image in multiple scales, generating a characteristic image, traversing the image in a sliding manner, and acquiring an RGB (red, green and blue) image and a depth image of a highway traffic sign; mapping the depth information of the depth image to an RGB image to generate a fusion image, and carrying out multi-scale scaling on the fusion image to generate a plurality of scaled images; calculating a feature image containing color information and gradient information of the zoomed image, calculating an integral image of the feature image, and generating a selection feature image by taking the features of the integral image as the input of a feature selection classifier; and traversing the selected characteristic image through the sliding window, and identifying the traffic sign in the sliding window. The invention also relates to a storage medium, electronic equipment and a highway sign identification system, and the invention adopts an automatic image identification mode to automatically judge the highway traffic sign, thereby improving the working efficiency and the identification precision of the traffic sign identification of vehicles in the running process of the highway.
The foregoing is merely a preferred embodiment of the invention and is not intended to limit the invention in any manner; those skilled in the art can readily practice the invention as shown and described in the drawings and detailed description herein; however, those skilled in the art should appreciate that they can readily use the disclosed conception and specific embodiments as a basis for designing or modifying other structures for carrying out the same purposes of the present invention without departing from the scope of the invention as defined by the appended claims; meanwhile, any changes, modifications, and evolutions of the equivalent changes of the above embodiments according to the actual techniques of the present invention are still within the protection scope of the technical solution of the present invention.