CN112017243B - Medium visibility recognition method - Google Patents

Abstract

The invention relates to the technical field of visibility recognition, and provides a medium visibility recognition method, which comprises the following steps: the method comprises the steps of collecting video data of a target object through a binocular camera, and collecting visibility data through a visibility tester; respectively extracting the positions of the target objects from two paths of video signals acquired by the binocular camera by using a target segmentation algorithm; performing feature matching on the obtained extraction result of the target object; obtaining distance information of a target object by using a binocular distance measuring algorithm; predicting the image quality visibility of each frame image in two paths of video signals acquired by the binocular camera by using an image quality prediction visibility algorithm; predicting the visibility of the target visual effect on each frame image in the two paths of video signals acquired by the binocular camera by using a target visual effect prediction visibility algorithm; and carrying out final visibility prediction by utilizing a visibility balance algorithm. The invention can improve the accuracy of medium visibility recognition and adapt to various environments.

Description

A medium visibility identification method

Technical Field

The present invention relates to the technical field of visibility recognition, and in particular to a medium visibility recognition method.

Background technique

Visibility identification is of great significance in aviation, navigation, transportation, etc. Severe weather conditions and marine environment will cause various safety hazards, which are related to the safety of people’s lives and property. If relevant departments can accurately release the corresponding visibility conditions, it can help all walks of life improve management quality.

Commonly used visibility identification methods include manual visual inspection and instrumental inspection. Manual visual inspection determines visibility by setting up special observation stations at each site. Visual inspection relies only on human eye discrimination and subjective judgment, resulting in poor standardization and objectivity. Instrumental inspection calculates visibility by measuring transmittance, extinction coefficient, etc. using equipment such as transmission visibility meters and laser radar visibility meters. These devices are expensive, have high site requirements, and have great limitations, so they cannot be widely used.

Summary of the invention

The present invention mainly solves the technical problems of the prior art of medium visibility recognition, such as high price, small application scope and low recognition accuracy, and proposes a medium visibility recognition method to achieve the purpose of improving the accuracy of medium visibility recognition and adapting to various environments.

The present invention provides a medium visibility identification method, comprising the following process:

Step 100, collecting video data of the target object through a binocular camera, and collecting visibility data through a visibility tester, to obtain two video signals and a visibility signal;

Step 200, using a target segmentation algorithm to extract the position of the target object from the two video signals collected by the binocular camera, and obtain the extraction result of the target object;

Step 300, performing feature matching on the target object extraction result;

Step 400, using a binocular distance measurement algorithm to obtain distance information of the target object, and then obtain the deviation between the detected distance of the target object and the actual distance;

Step 500, using an image quality prediction visibility algorithm to perform image quality visibility prediction on each frame of the two-channel video signal collected by the binocular camera, and predict the visibility range;

Step 600, using a target visual effect prediction visibility algorithm to predict the target visual effect visibility for each frame of the two-channel video signal collected by the binocular camera, and predicting the visibility interval;

Step 700, using the visibility balance algorithm to perform final visibility prediction.

Further, step 200 includes the following process:

Step 201, performing a convolutional neural network to extract features for each frame of the two video signals;

Step 202, using a region extraction network to perform preliminary classification and regression;

Step 203, aligning the candidate frame feature map;

Step 204, using a convolutional neural network to classify, regress, and segment the target to obtain an extraction result of the target object.

Further, step 300 includes the following process:

Step 301, extract key points from the contours of two targets;

Step 302, positioning the key points obtained;

Step 303, determining the feature vector of the key point according to the located key point;

Step 304: matching key points using the feature vectors of each key point.

Further, step 400 includes the following process:

Step 401, calibrating the binocular camera;

Step 402, performing binocular calibration on the binocular camera;

Step 403, performing binocular matching on the images collected by the binocular camera;

Step 404: Calculate the depth information of the image after binocular matching to obtain the distance information of the target object in the image.

Further, step 500 includes the following process:

Step 501, segmenting the image to identify and locate the target;

Step 502: predict the image visibility based on the target recognition and positioning results to obtain the image classification result.

Further, step 600 includes the following process:

Step 601, constructing a target visual effect prediction visibility algorithm network structure;

Step 602, inputting the target object extraction result obtained in step 200 into the target visual effect prediction visibility algorithm network structure to obtain a multi-scale feature map;

Step 603, classify the image through the target visual effect prediction visibility algorithm network structure to obtain the target image classification result and realize the predicted visibility range.

Furthermore, the network structure of the target visual effect prediction visibility algorithm includes: an input layer, a convolution layer, a first feature extraction module, a merging channel, a second feature extraction module, a merging channel, a fully connected layer, and a classification structure output layer; wherein each feature extraction module includes 5 convolution kernels.

Further, step 700 includes the following process:

Step 701, constructing a visibility balance algorithm network structure, wherein the visibility balance algorithm network structure includes an input layer, a recurrent neural network, a fully connected layer, and a visibility interval output layer;

Step 702, input the visibility into the recurrent neural network in sequence to obtain a result considering the time series;

Step 703, connect the output of the recurrent neural network to a fully connected layer to obtain the visibility interval value corresponding to the time series.

The present invention provides a medium visibility identification method, which uses a binocular camera to capture video images around the clock, uses the deviation between the target detection distance and the actual distance obtained by the ranging algorithm, uses the visibility interval obtained by the image quality prediction visibility algorithm, uses the visibility interval obtained by the target visual effect prediction visibility algorithm, and uses the visibility balance algorithm to perform final visibility prediction based on the above results. The present invention can identify the current medium visibility, has high accuracy and stability in medium visibility identification, is highly adaptable to various common situations, and does not rely on specific video acquisition equipment. Each point of the present invention uses a binocular camera to collect video data. The use of the binocular camera achieves multiple goals at one stroke. The two lenses can be used independently, each lens can be used as an independent video signal source, and the two signals can be cross-validated, and can also be used in combination to increase sensitivity to distance.

The present invention can be applied to seabed visibility identification, atmospheric visibility identification in port areas, and other scenarios that require medium visibility identification. In the atmospheric visibility identification in the port area, through the analysis of the application scenarios in the port area, it can be found that the port area is large and the operating area is widely distributed. Therefore, the identification points need to be deployed at multiple points according to the operating area. The construction in the port area is relatively mature, and the topographic features and building appearance are relatively stable. It is convenient to set detection reference points at each point, which is more conducive to improving the stability and accuracy of identification. The binocular camera is deployed at multiple points in the port area. Through the system's timestamp control, video data of each point at the same time can be obtained. At the same time, the video data is an image sequence in the time dimension. Therefore, the atmospheric visibility data of different time periods and different locations can be obtained by the method of the present invention for use by port business personnel.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a flow chart of a method for identifying medium visibility provided by the present invention;

FIG2 is a schematic diagram of a feature pyramid network structure;

FIG3 is a schematic diagram of a bottom-up structure;

4a-4e are schematic diagrams showing the principle of generating feature maps at each stage in the bottom-up structure;

FIG5 is a schematic diagram of a region extraction network structure;

FIG6 is a diagram showing the effect of performing an alignment operation on a feature map;

Figure 7 is a schematic diagram of the classification, regression, and segmentation network structures;

FIG8 is a schematic diagram of the principle of a binocular ranging algorithm;

FIG9 is a schematic diagram of the basic principle of binocular ranging;

FIG10 is a schematic diagram of an image segmentation network structure;

FIG11 is a schematic diagram of an image visibility prediction network structure;

FIG12 is a schematic diagram of a network structure of a target visual effect prediction visibility algorithm;

FIG13 is a schematic diagram of the visibility balance algorithm network structure;

14a-14b are schematic diagrams of a recurrent neural network structure.

Detailed ways

In order to make the technical problems solved by the present invention, the technical solutions adopted and the technical effects achieved clearer, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It is understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for the convenience of description, only the parts related to the present invention are shown in the accompanying drawings, rather than all the contents.

FIG1 is a flowchart of the method for identifying medium visibility provided by the present invention. As shown in FIG1 , the method for identifying medium visibility provided by an embodiment of the present invention includes:

Step 100, collecting video data of the target object through a binocular camera, and collecting visibility data through a visibility tester, to obtain two video signals and a visibility signal.

Since the output of visibility recognition of the present invention is a discrete value, that is, a digital interval, such as the description of "below 500 meters", "500 meters to 1000 meters". Therefore, in order to improve the detection accuracy, the continuous value obtained by the ranging algorithm, that is, the detection distance, such as the description of "245.87 meters", "1835.64 meters", is used to correct the discrete values detected by other algorithms. Therefore, it is necessary to set a target reference object. The selection principles of the target reference object: a fixed object with unchanged position; an object that can be clearly identified both day and night under good visibility; no obstruction between the binocular camera and the target object; the distance between the target object and the binocular camera conforms to the distribution of the visibility interval and is evenly distributed. A distance difference of about 100 meters is appropriate.

Step 200, using a target segmentation algorithm to extract the position of the target object from two video signals collected by the binocular camera, to obtain the extraction result of the target object.

The position of the target object is extracted from the two video signals of the binocular camera respectively to prevent the subsequent calculation failure caused by errors in single-channel detection. This step uses an accurate target segmentation algorithm to obtain the precise outline of the target object, which can extract the accurate position of the target object and prepare for subsequent processing.

Considering that the field of view (angle and focal length, etc.) of the binocular camera will not change easily, in theory, the position of the target in the image is fixed. However, in practice, it must be considered that the camera will shake under the influence of wind and waves, or other external forces will cause slight changes in the field of view, and even interference objects such as flying birds and schools of fish will appear in the field of view. In order to increase the accuracy of detection, when the target segmentation algorithm is processed, the hotspot area in the field of view will be set according to the prior conditions, and the weight of the target detected in the hotspot area will be increased.

After the target segmentation algorithm, in the two video frame signals of the binocular camera, the "precise contours" of the two targets can be theoretically obtained. The "precise contours" mentioned here will be affected by different conditions in the medium. Under different visibility conditions, the "precise contours" detected may be different. We tolerate this interference in this link because the interference contains visibility information. If two "precise contours" are not obtained here, it means that the frame data is incorrectly recognized, or the recognition cannot be performed normally due to some reasons, such as abnormal acquisition of a recorded video signal, or a lens is blocked.

If the above conditions are not met, the frame data will be discarded and the next frame data will be input and re-identified. In actual application of this method, if the above situation occurs for multiple frames in succession, an alarm needs to be issued and the video signal needs to be saved for inspection by business personnel.

The target segmentation in this step is the first step of image analysis, the foundation of computer vision, an important part of image understanding, and one of the most difficult problems in image processing. Image segmentation refers to dividing an image into several non-intersecting areas based on features such as grayscale, color, spatial texture, and geometric shape, so that these features show consistency or similarity in the same area, but show obvious differences between different areas. Simply put, it is to separate the target from the background in an image. Image segmentation greatly reduces the amount of data to be processed in subsequent advanced processing stages such as image analysis and target recognition, while retaining information about the structural features of the image.

The target segmentation algorithm is mainly divided into: threshold-based segmentation method, region-based segmentation method, edge-based segmentation method and deep learning-based segmentation method. The main process of the target segmentation algorithm used in this step includes:

Step 201, for each frame of the two video signals, a convolutional neural network is used to extract features.

This step considers that the clarity of the image will change with different camera parameters, so a multi-scale feature extraction solution is adopted, namely the feature pyramid network. The structure of the feature pyramid network is shown in Figure 2.

The feature pyramid network is divided into two parts. The structure on the left is called the bottom-up structure, which produces feature maps of different scales, such as C1 to C5 in the figure. C1 to C5 are feature maps of different scales. From bottom to top, the size of the feature map keeps getting smaller, which also means that the extracted feature dimensions are getting higher and higher. The shape is pyramid-shaped, so it is called a feature pyramid network. The structure on the right is called the top-down structure, which corresponds to each layer of the feature pyramid. Between the two structures, the arrows connecting the feature processing of the same level are horizontal connections.

The purpose of doing this is that smaller high-level features have more semantic information, while larger low-level features have less semantic information but more position information. Through such a connection, the feature map of each layer integrates features of different resolutions and different semantic strengths. Therefore, when detecting objects of different resolutions, the detection effect can be improved.

The bottom-up structure is shown in Figure 3. The network structure contains five stages, each of which is used to calculate feature maps of different sizes, with a scaling step of 2. The principle of generating feature maps in each stage is shown in Figures 4a-4e. We use the C1, C2, C3, C4, and C5 feature maps output by each stage to construct the feature pyramid network structure.

The top-down structure is shown on the right side of the pyramid network structure in Figure 2. First, the high-level feature maps with stronger semantic information are upsampled to the same size as the low-level feature maps. Then, the feature maps in the bottom-up and top-down structures with the same size are horizontally connected. The two feature map maps are merged in an element-wise manner. Finally, in order to reduce the aliasing effect caused by upsampling, a convolutional layer is attached to each merged feature map to obtain the final feature map, namely P2, P3, P4, and P5.

Step 202: Perform preliminary classification and regression using a region extraction network.

The structure of the region extraction network is shown in Figure 5. Based on the feature maps P2, P3, P4, and P5 obtained by the feature pyramid network, first generate anchor frames corresponding to the original image for each point on the feature map according to the anchor frame generation rules, and then input the P2, P3, P4, and P5 feature maps into the region extraction network. The region extraction network contains a convolution layer and a fully connected layer, and finally obtains the classification and regression results of each anchor frame, including the foreground and background classification scores of each anchor frame and the bounding box coordinate correction information of each anchor frame. Finally, the anchor frame that meets the foreground score condition is selected according to the threshold and the bounding box is corrected. The corrected anchor frame is called a candidate frame.

Step 203: align the candidate box feature map.

Through the region extraction network, candidate boxes that meet the score requirements are obtained and mapped back to the feature map. The number of feature layers corresponding to the candidate boxes is obtained according to the following formula:

Among them, w represents the width of the candidate box, h represents the height of the candidate box, and k represents the number of feature layers corresponding to the candidate box. k ₀ is the number of layers mapped when w and h = 224, which is generally 4, corresponding to the P4 layer. Then, the feature map corresponding to the candidate box is obtained by bilinear interpolation, and the size of the obtained feature map is consistent. The effect of aligning the feature map is shown in Figure 6.

Step 204, using a convolutional neural network to classify, regress, and segment the target to obtain an extraction result of the target object.

The classification, regression, and segmentation network structures are shown in Figure 7. Based on the fixed-size candidate box feature map obtained above, the classification score and coordinate offset of the candidate box are calculated through the classification and regression network, and the candidate box is corrected by the bounding box. After the segmentation network, the target in the candidate box is segmented. Finally, the classification, bounding box regression, and segmentation results of the target in the image can be obtained through the target segmentation algorithm, and then the target object extraction result can be obtained.

Step 300, performing feature matching on the target object extraction result.

Through the target segmentation algorithm of step 200, two target contours are obtained, but the positions and angles of the two target contours in different video frames are different. This step requires feature matching of the two target contours. The feature matching algorithm needs to perform feature comparison on the two target contours to find the same point of the same object at different positions in the imaging. Because the subsequent ranging algorithm must be calculated based on a certain pixel point. In this link, in order to ensure that the same point is extracted as much as possible, multiple sampling and averaging methods will be used to determine the final result. And record the pixel position of the point in different imaging. Specifically include the following processes:

Step 301: extract key points from the contours of two targets.

Key points are some very prominent points that will not disappear due to factors such as lighting, scale, rotation, etc., such as corner points, edge points, bright spots in dark areas, and dark spots in bright areas. This step searches for image locations in all scale spaces. The Gaussian differential function is used to identify potential scale- and rotation-invariant points of interest.

Step 302: locate the key points obtained.

At each candidate location, a fine-grained model is fitted to determine the position and scale. Keypoints are selected based on their stability.

Step 303: Determine the feature vector of the key point based on the located key point.

Based on the local gradient direction of the image, one or more directions are assigned to each keypoint position. All subsequent operations on the image data are transformed relative to the direction, scale and position of the keypoint, thereby providing invariance to these transformations.

Step 304: matching key points using the feature vectors of each key point.

Through the feature vectors of each key point, we can compare them two by two to find several pairs of feature points that match each other, and establish the corresponding relationship between the features of the objects. Finally, we can calculate the distance between the key points through the corresponding relationship.

Step 400, using a binocular distance measurement algorithm, obtains distance information of the target object, and then obtains the deviation between the detected distance of the target object and the actual distance.

The principle diagram of the binocular ranging algorithm is shown in Figure 8. As can be seen from Figure 8, the error of the ranging algorithm will be affected by factors such as the measurement error of the distance between the left and right cameras, the measurement error of the camera focal length, and the measurement error of the vertical height difference between the camera and the target. These errors are inevitable. However, this step is not to measure the precise distance of the target, but to establish the correlation between the actual distance and the detection distance under different visibility conditions. In addition, due to the existence of the subsequent neural network, the error generated in this step can be reduced through the subsequent neural network. The output value of the ranging algorithm is the detection distance value (continuous value). The basic principle of binocular ranging is shown in Figure 9. This step specifically includes the following processes:

Step 401, calibrate the binocular camera.

Due to the characteristics of the optical lens of the camera, the imaging has radial distortion, which can be determined by three parameters k1, k2, and k3. The radial distortion formula is: _Xdr = X(1+ _k1 × ^r2 + _k2 × ^r4 + _k3 × ^r6 ), _Ydr = Y(1+ _k1 × ^r2 + ^k2 _× r4+ _k3 × ^r6 ), ^r2 = ^X2 + ^Y2 , where (X, Y) are the coordinates of the pixel point of the undistorted image, and ( _Xdr , _Ydr ) are the coordinates of the pixel point of the distorted image. Due to the assembly error, the sensor and the optical lens of the camera are not completely parallel, so the imaging has tangential distortion, which can be determined by two parameters p1 and p2. The tangential distortion formula is: _Xdt = _2p1 ×X×Y+ _p2 ( ^r2 +2× ^X2 )+1, _Ydt = _2p1 ( ^r2 +2×Y ² )+2p ₂ ×X×Y+1, where (X,Y) are the coordinates of the pixels in the undistorted image, and (X _dt ,Y _dt ) are the coordinates of the pixels in the distorted image. The calibration of a single camera mainly involves calculating the camera's internal parameters (focal length f and imaging origin cx, cy, five distortion parameters (generally only k1, k2, p1, p2 need to be calculated, and k3 only needs to be calculated for fisheye lenses with particularly large radial distortion)) and external parameters (world coordinates of the calibration object). The calibration of a binocular camera not only requires the internal parameters of each camera, but also requires the measurement of the relative position between the two cameras through calibration (i.e. the rotation matrix R and translation vector t of the right camera relative to the left camera).

Step 402: Perform binocular calibration on the binocular camera.

Binocular correction is to eliminate distortion and align the left and right views based on the monocular internal parameter data (focal length, imaging origin, distortion coefficient) and the binocular relative position relationship (rotation matrix and translation vector) obtained after camera calibration, so that the imaging origin coordinates of the left and right views are consistent, the optical axes of the two cameras are parallel, the left and right imaging planes are coplanar, and the epipolar lines are aligned. In this way, any point on one image and its corresponding point on the other image must have the same row number, and the corresponding point can be matched by performing a one-dimensional search in the row.

Step 403: perform binocular matching on the images captured by the binocular camera.

The purpose of binocular matching is to match the corresponding image points of the same scene in the left and right views in order to obtain disparity data.

Step 404: Calculate the depth information of the image after binocular matching to obtain the distance information of the target object in the image.

P is a point on the object to be measured, L and R are the optical centers of the left and right cameras respectively, and the imaging points of point P on the two camera sensors are p and p' respectively (the imaging plane of the camera is rotated and placed in front of the lens), f is the focal length of the camera, b is the center distance between the two cameras, and z is the distance of the target object. Let the distance between p and p' be dis, then

dis＝b-(X _R -X _L )

According to the triangle similarity principle:

Available:

In the formula, the focal length f and the camera center distance b can be obtained through calibration, so as long as the value of X _R - _XL (i.e., disparity d) is obtained, the depth information can be obtained. The disparity value can be calculated based on the matching key points in the second feature matching algorithm. Finally, the distance information of the target object in the image can be obtained through the binocular ranging algorithm, and then the deviation between the detected distance of the target object and the actual distance can be obtained.

Step 500, using an image quality prediction visibility algorithm to perform image quality visibility prediction on each frame of the two-channel video signal collected by the binocular camera, and predict the visibility range.

The image quality prediction visibility algorithm is a method that uses the macro information of the image to predict visibility. It mainly predicts visibility based on the clarity and contrast of objects in the medium background of the image. This link cannot directly receive the video frame signal, but needs to filter out the close-up high-frequency information in the image, extract the low-frequency information of the medium background, and then analyze and predict the image. In order to make the present invention adaptable to day and night, and improve the prediction accuracy under different conditions, a large amount of video data and detection data of the visibility detector with the same timestamp need to be provided during the training process of the algorithm. The output of this step is the interval value (discrete value) of visibility. This step is mainly divided into the following two steps:

Step 501, segment the image to identify and locate the target.

The image segmentation network structure is shown in Figure 10. The image is subjected to three convolution blocks for feature extraction, and then connected to two fully connected layers to obtain the classification score and bounding box position of the target in the image. Finally, the highest score is selected as the output, and the bounding box where the target is most likely to exist is extracted, thereby achieving the purpose of identifying and locating the target.

Step 502: predict the image visibility based on the target recognition and positioning results to obtain the image classification result.

The network structure of image visibility prediction is shown in Figure 11. Based on the above image segmentation results, the predicted visibility image can be obtained. Since the image scene is relatively complex, the network structure contains three modules. Each module uses four different convolution kernels to extract features of different scales of the image, which increases feature diversity and improves classification accuracy. At the output end of each module, the various extracted features are spliced and merged in the channel dimension to obtain a multi-scale feature map. Finally, the image is classified through the fully connected layer to obtain the image classification result and realize the predicted visibility range.

Step 600, using a target visual effect prediction visibility algorithm to predict the target visual effect visibility for each frame of the two-channel video signal collected by the binocular camera, and predicting the visibility range.

The target visual effect prediction visibility algorithm is a method that uses image microscopic information to predict visibility, mainly based on the target object's contour gradient, contour completeness, and color saturation to predict visibility. The input of this link algorithm is the output of the target segmentation algorithm of step 200. In order to make the present invention adaptable to daytime and nighttime, and to improve the prediction accuracy under different conditions, a large amount of video data and detection data of the visibility detector with the same timestamp need to be provided during the training process of the algorithm. The output of this step is the interval value (discrete value) of visibility. Step 600 specifically includes the following process:

Step 601, constructing a target visual effect prediction visibility algorithm network structure.

The network structure of the target visual effect prediction visibility algorithm is shown in Figure 12. The network structure of the target visual effect prediction visibility algorithm includes: an input layer, a convolution layer, a first feature extraction module, a merging channel, a second feature extraction module, a merging channel, a fully connected layer, and a classification structure output layer; each feature extraction module includes 5 convolution kernels. Based on the target segmentation algorithm of step 200, an image containing a target can be obtained. Since the environmental noise interference in the target image is relatively small, the network structure constructed in this step includes two feature extraction modules, each of which uses three different convolution kernels to extract features of different scales of the image, thereby increasing feature diversity and improving classification accuracy.

In step 602, the extraction result of the target object obtained in step 200 is input into the target visual effect prediction visibility algorithm network structure to obtain a multi-scale feature map.

Specifically, in this step, the various features extracted are concatenated and merged in the channel dimension at the output end of each module to obtain a multi-scale feature map.

Step 603, classify the image through the target visual effect prediction visibility algorithm network structure to obtain the target image classification result and realize the predicted visibility range.

Specifically, in this step, the image is classified through the fully connected layer to obtain the classification result of the target image.

Step 700, using the visibility balance algorithm to perform final visibility prediction.

After the processing from step 200 to step 600, three results related to visibility are obtained in one frame of video data: (1) The deviation (continuous value) between the detected distance of the target object obtained by the distance measurement algorithm in step 400 and the actual distance. The generation of this deviation is greatly and directly related to visibility. (2) The visibility interval (discrete value) obtained by the image quality prediction visibility algorithm in step 500. (3) The visibility interval (discrete value) obtained by the target visual effect prediction visibility algorithm in step 600.

Traditionally, the balancing strategy for multiple calculation results usually adopts the method of directly taking the average, or taking the average after excluding outliers. In order to further improve the detection accuracy, a multi-frame result cyclic verification method is adopted in this step. In a short time compared with the visibility change speed (such as 1 minute), multiple frames of data are obtained at a certain time interval (such as 5 seconds) for calculation, and the detection results of each frame are input into the visibility balance algorithm in chronological order to obtain the final visibility interval value. Step 700 includes the following process:

Step 701, constructing a visibility balance algorithm network structure, wherein the visibility balance algorithm network structure includes an input layer, a recurrent neural network, a fully connected layer, and a visibility interval output layer.

The network structure of the visibility balance algorithm is shown in FIG13. As shown in FIG13, the network structure of the visibility balance algorithm includes an input layer, a recurrent neural network, a fully connected layer, and a visibility interval output layer. The visibility balance algorithm network inputs visibility in chronological order, and the visibility feature length input at each time node is 3, which are respectively the deviation between the target object detection distance and the actual distance obtained by the ranging algorithm in step 400, the visibility interval obtained by the image quality prediction visibility algorithm in step 500, and the visibility interval obtained by the target visual effect prediction visibility algorithm in step 600.

Step 702, input the visibility into the recurrent neural network in sequence to obtain a result that takes into account the time series.

The present invention utilizes a visibility balance algorithm, which can balance multiple calculation results in the time dimension and reduce the impact of single-frame calculation errors. However, the results obtained at different timestamps have a certain correlation before and after, that is, the visibility will not change drastically in a short period of time. Therefore, the time dimension can be used to correct multiple detection values. Therefore, visibility balance is first processed using a recurrent neural network. The characteristic of a recurrent neural network is that each calculation needs to consider the result of the previous calculation as a priori input, which can achieve the effect of correcting subsequent calculations. After correction, the calculation results of different timestamps are obtained, and then a fully connected neural network is connected to integrate the multiple calculation results to obtain the final result. The recurrent neural network structure is shown in Figures 14a-14b. The recurrent neural network structure includes: an input layer, a circulation layer, and an output layer.

Recurrent neural networks have the characteristic of learning recursively according to the order of input data, so they can be used to process sequence-related data. From the network structure, it can be seen that recurrent neural networks will remember previous information and use previous information to affect the output of subsequent nodes. In other words, the nodes between the hidden layers of the recurrent neural network are connected, and the input of the hidden layer includes not only the output of the input layer, but also the output of the hidden layer at the previous moment.

Given the data X={X ₁ ,X ₂ ,…,X _t } input in sequence, the feature length of X is c and the expansion length is t. The output h _t of the recurrent neural network is calculated as:

h _t = tanh(W*X _t +W*h _t-1 )

Among them, W is the hidden layer parameter, and tanh is the activation function. It can be seen from the formula that the output at time t depends not only on the input X _t at the current time, but also on the output h _t-1 at the previous time.

Step 703, connect the output of the recurrent neural network to a fully connected layer to obtain the visibility interval value corresponding to the time series.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that modifying the technical solutions described in the aforementioned embodiments, or replacing some or all of the technical features therein by equivalents, does not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims (3)

1. A method for identifying visibility of a medium, comprising the steps of:

step 100, acquiring video data of a target object through a binocular camera, and acquiring visibility data through a visibility tester to obtain two paths of video signals and a visibility signal;

Step 200, respectively extracting the positions of the target objects from two paths of video signals acquired by the binocular camera by using a target segmentation algorithm to obtain extraction results of the target objects;

Step 300, performing feature matching on the obtained extraction result of the target object;

Step 400, obtaining the distance information of the target object by using a binocular distance measuring algorithm, and further obtaining the deviation between the detection distance of the target object and the actual distance; step 400 includes the following steps 401 to 404:

step 401, calibrating a binocular camera;

Step 402, performing binocular correction on a binocular camera;

Step 403, performing binocular matching on the image acquired by the binocular camera;

Step 404, calculating depth information of the binocular-matched image to obtain distance information of a target object in the image;

step 500, predicting the image quality visibility of each frame image in two paths of video signals acquired by a binocular camera by using an image quality prediction visibility algorithm, and predicting the obtained visibility interval; step 500 includes the following steps 501 to 502:

Step 501, dividing an image to realize identification and positioning of a target;

step 502, predicting the visibility of the image according to the identification and positioning results of the target to obtain an image classification result;

step 600, predicting the visibility of the target visual effect for each frame image in two paths of video signals acquired by the binocular camera by using a target visual effect prediction visibility algorithm, and predicting the obtained visibility interval; step 600 includes the following steps 601 to 603:

Step 601, constructing a target visual effect prediction visibility algorithm network structure; the target visual effect prediction visibility algorithm network structure comprises: the system comprises an input layer, a convolution layer, a first extraction feature module, a merging channel, a second extraction feature module, a merging channel, a full connection layer and a classification structure output layer; wherein each extracted feature module comprises 5 convolution kernels;

step 602, inputting the extraction result of the target object obtained in the step 200 into a network structure of a target visual effect prediction visibility algorithm to obtain a multi-scale feature map;

step 603, classifying images through a target visual effect prediction visibility algorithm network structure to obtain a target image classification result, and realizing a predicted visibility interval;

Step 700, performing final visibility prediction by using a visibility balance algorithm; step 700 includes the following steps 701 through 703:

Step 701, constructing a visibility balance algorithm network structure, wherein the visibility balance algorithm network structure comprises an input layer, a circulating neural network, a full-connection layer and a visibility interval output layer;

Step 702, sequentially inputting the visibility into a cyclic neural network to obtain a result considering a time sequence;

And step 703, connecting the output of the cyclic neural network with a full-connection layer to obtain a visibility interval value corresponding to the time sequence.

2. The method of medium visibility identification according to claim 1, wherein step 200 comprises the following process:

Step 201, convolutional neural network extraction features are performed on each frame of image in two paths of video signals;

Step 202, performing preliminary classification and regression by using a region extraction network;

step 203, performing alignment operation on the candidate frame feature map;

And 204, classifying, regressing and dividing the target by using the convolutional neural network to obtain an extraction result of the target object.

3. The medium visibility identification method according to claim 1, wherein step 300 comprises the following procedure:

Step 301, extracting key points from two target contours;

step 302, positioning the key points;

Step 303, determining feature vectors of the key points according to the positioned key points;

step 304, the key points are matched through the feature vectors of the key points.