CN110675449B - A rip current detection method based on binocular camera - Google Patents

Abstract

The invention discloses an offshore current detection method based on a binocular camera, which belongs to the field of computer vision and the technical field of security assurance. The method of the present invention includes the following steps: first, training a convolutional neural network algorithm for identifying offshore currents secondly, in the actual detection process, the trained convolutional neural network is used to identify whether there is an rip current on the ocean wave image collected by the binocular camera; finally, the identified rip current is located; the method of the present invention can The 3D information of the rip current is obtained based on binocular stereo vision; the more accurate position information of the rip current can be obtained in time, the method is simple, and the operability is strong. Signs to remind visitors to reduce drowning incidents due to rip currents.

Description

A rip current detection method based on binocular camera

technical field

The invention belongs to the field of computer vision and the technical field of security assurance, and in particular relates to a method for detecting an offshore current based on a binocular camera.

Background technique

The rip current velocity is mostly 0.3-1 meters per second, the fastest can reach 3 meters per second, and its length can reach 30-100 meters or even longer, and the flow direction is almost perpendicular to the shoreline, which can quickly drag strong swimmers into it. Deep water, causing drowning. Rip current has become another marine disaster that causes harm to people's coastal tourism after storm surge and ocean waves. About 90% of seaside drownings are caused by rip currents. The rip current has brought a lot of problems to the attractive maintenance of coastal tourism, beach management, and handling of accidents and disputes, which has seriously affected the healthy development of coastal tourism economy. At present, the technical evaluation and safety management of rip current disasters in my country have just started, and the relevant investigation and evaluation, risk assessment, refined forecast, safety management and public science warning are extremely lacking; the public's understanding of rip current also has blind spots. Misunderstandings, cognitive errors and lack of vigilance have resulted in a large number of drowning incidents in many hot tourist areas, increasing the rescue workload and making coastal tourism safety management difficult. Therefore, there is a great need for an efficient and simple method for rip current detection. At present, the traditional detection method for rip current is to place buoys or current meters near the shore.

SUMMARY OF THE INVENTION

In view of the above technical problems existing in the prior art, the present invention proposes an offshore current detection method based on a binocular camera, which has a reasonable design, overcomes the deficiencies of the prior art, and has good effects.

In order to achieve the above object, the present invention adopts the following technical solutions:

A rip current detection method based on a binocular camera, comprising the following steps:

Step 1: Collect the image set for training the convolutional neural network, and train the convolutional neural network;

Step 2: Use the image recognition algorithm of the trained convolutional neural network to identify the rip current on the collected image; determine whether the rip current exists;

If: the judgment result is that the rip current exists, then find the feature points of the rip current, and extract the feature points of the rip current; the feature points include the midpoint and the left and right edge points of the rip current;

Or the judgment result is that the rip current does not exist, then the image is collected again for processing;

Step 3: Use the binocular positioning algorithm to locate the identified feature points of the rip current, and use the position of the feature points of the rip current to identify the position of the rip current;

Obtaining the three-dimensional coordinates of the rip current based on binocular stereo vision technology;

Step 4: Based on the location information of the feature points of the rip current, the dangerous area is divided and fed back to the relevant staff, and the staff reminds tourists by marking.

Preferably, in step 2, the binocular cameras include left and right cameras. Since the data collected by the left and right cameras are basically the same, the image of the single-sided camera is used for rip current identification.

Preferably, the specific steps of training the convolutional neural network in step 1 are as follows:

Step 1.1: Image data preprocessing; specifically includes the following steps:

Step 1.1.1: Process the training data collected in step 1; convert the collected training data into a data format that TensorFlow can recognize;

Step 1.1.2: Add a label according to the image, put the image and label into the array, and convert the array into a format that Tensorflow can recognize;

Step 1.1.3: Normalize the image including cropping and supplementing;

Step 1.2: Build a convolutional neural network model based on the Tensorflow framework;

The classic convolutional neural network LeNet-5 model is adopted; the model is divided into 7 layers: convolutional layer - pooling layer - convolutional layer - pooling layer - fully connected layer - fully connected layer - fully connected output layer; among them, The convolutional layer extracts the preliminary offshore current features, the pooling layer extracts the main features of the offshore current, and the fully connected layer summarizes the features of each part;

Step 1.3: Use the constructed convolutional neural network model to train the neural network for identifying rip currents.

Preferably, in step 3, it specifically includes the following steps:

Step 3.1: Calibrate the left and right cameras;

The camera is calibrated by Zhang Zhengyou's calibration method, and the internal and external parameters of the left and right cameras are obtained;

Step 3.2: Correct the left and right camera images;

Use the internal and external parameters obtained by the calibrated left and right cameras to perform distortion correction and stereo correction on the image;

Step 3.3: Stereo matching the image;

Use the SGBM algorithm to perform stereo matching on the image, and finally obtain the disparity map;

Step 3.4: Obtain the three-dimensional coordinate information of the rip current;

Using the disparity map and the internal parameters of the left and right cameras, the depth image is obtained, and the three-dimensional coordinates of the rip current feature points are obtained according to the camera model and the parameters obtained by the calibration camera;

Step 3.5: Take the position information of the midpoint of the rip current as the position of the rip current, and divide the dangerous area at the left and right edge points of the obtained rip current, that is, according to the nature of the rip current in the area of the rip current, according to the actual situation Expand to get the danger zone.

Preferably, in step 3.3, SGBM is a semi-global block matching algorithm, and the SGBM algorithm specifically includes the following steps:

S1: image preprocessing;

Use the horizontal Sobel operator to process the image, map the pixels to obtain a new image, and preprocess the gradient information of the original image;

S2: cost calculation;

The cost calculation is divided into two steps: one is the gradient cost calculation of the gradient information of the preprocessed image obtained by the sampling-based method; the second is the SAD cost calculation of the original image obtained by the sampling-based method;

S3: dynamic programming;

Accumulate energy in each direction according to the idea of dynamic programming, and then add the matching cost in each direction to get the total matching cost;

S4: post-processing;

The post-processing part needs to perform uniqueness detection, sub-pixel interpolation and left-right consistency detection.

Beneficial technical effects brought by the present invention:

The invention collects images through binocular cameras, uses an image recognition algorithm based on a convolutional neural network to identify offshore currents, uses the principle of binocular positioning to locate offshore currents, and provides a new method for offshore current detection; The method based on the binocular camera to detect the offshore current can obtain more accurate offshore current position information in time. The method is simple, the labor cost is low, and the operability is strong. The location information of the stream marks the location of the beach, reminds tourists, reduces the occurrence of drowning events caused by rip currents, and provides a practical approach for coastal tourism management.

Description of drawings

Fig. 1 is a flow chart of the realization of a method for rip current detection based on a binocular camera;

Fig. 2 is the flow chart of binocular positioning algorithm;

Fig. 3 is the flow chart of recognition image algorithm;

Figure 4 is a schematic diagram of an offshore rip current model.

Detailed ways

The present invention is described in further detail below in conjunction with the accompanying drawings and specific embodiments:

As shown in Figure 1, a method for detecting rip current based on a binocular camera includes the following steps:

Step 1: Collect the image set for training the convolutional neural network, and train the convolutional neural network.

Among them, the training data is to provide a basis for subsequent image recognition, which can be the nearshore waves collected by the binocular camera, and the image can have either rip current or no rip current. The nearshore wave image data with and without the rip current that exists on the Internet; it can also be the nearshore wave image data of the ideal model of the rip current generated by the computer.

The specific steps for training the convolutional neural network in step 1 are as follows:

Step 1.1: Image data preprocessing; specifically includes the following steps:

Step 1.1.1: Process the training data collected in step 1, and convert the collected training data into a data format that TensorFlow can recognize;

Step 1.1.2: Add a label according to the image, put the image and label into the array, and convert the array into a format that Tensorflow can recognize;

Step 1.1.3: Normalize the image including cropping and supplementing;

Step 1.2: Build a convolutional neural network model based on the Tensorflow framework;

The classic convolutional neural network LeNet-5 model is adopted; the model is divided into 7 layers: convolutional layer - pooling layer - convolutional layer - pooling layer - fully connected layer - fully connected layer - fully connected output layer; among them, The convolutional layer extracts the preliminary offshore current features, the pooling layer extracts the main features of the offshore current, and the fully connected layer summarizes the features of each part;

Step 1.3: Use the constructed convolutional neural network model to train the neural network for identifying rip currents.

Step 2: After the training is completed, carry out the experimental operation, and the collection of actual ocean waves is called actual data, and the actual data is the images collected on the ocean waves during the actual rip current observation. Using the image recognition algorithm of the trained CNN convolutional neural network (the process is shown in Figure 2), the collected images are used to identify the rip current; and determine whether the rip current exists;

If: the judgment result is that the rip current exists, then find the feature points of the rip current; the feature points include the midpoint and the left and right edge points of the rip current;

Or the judgment result is that the rip current does not exist, then the image is collected again for processing;

The binocular camera includes left and right cameras. Since the data collected by the left and right cameras are basically the same, the single-sided camera image is used for rip current identification.

Step 3: Use the binocular positioning algorithm (the process of which is shown in Figure 3) to locate the identified feature points of the rip current, and use the position of the feature point of the rip current to identify the position of the rip current;

Obtaining the three-dimensional coordinates of the rip current based on binocular stereo vision technology;

Specifically include the following steps:

Step 3.1: Calibrate the left and right cameras;

The camera is calibrated by Zhang Zhengyou's calibration method, and the internal and external parameters of the left and right cameras are obtained; Zhang Zhengyou's calibration method is a camera calibration method based on a moving plane template. . Overcome the shortcomings of both and combine the advantages of both. The specific steps include calculating a homography matrix, calculating an internal parameter matrix, calculating an external parameter matrix, and calculating a distortion parameter.

Step 3.2: Correct the left and right camera images;

Use the internal and external parameters obtained by the calibrated left and right cameras to perform distortion correction and stereo correction on the image;

Step 3.3: Stereo matching the image;

The SGBM algorithm is used to perform stereo matching on the image to obtain the disparity map, and finally the actual three-dimensional coordinate information of the point in the image is obtained. SGBM is a semi-global block matching algorithm, which has the characteristics of good parallax effect and fast speed; the steps of the SGBM algorithm are as follows:

S1: image preprocessing;

Use the horizontal Sobel operator to process the image, map the pixels to obtain a new image, and preprocess the gradient information of the original image;

S2: cost calculation;

The cost calculation is divided into two steps: one is the gradient cost calculation of the gradient information of the preprocessed image obtained by the sampling-based method; the second is the SAD cost calculation of the original image obtained by the sampling-based method;

S3: dynamic programming;

Accumulate energy in each direction according to the idea of dynamic programming, and then add the matching cost in each direction to get the total matching cost;

S4: post-processing;

The post-processing part needs to perform uniqueness detection, sub-pixel interpolation and left-right consistency detection.

Step 3.4: Obtain the three-dimensional coordinates of the rip current;

Using the disparity map and the internal parameters of the left and right cameras, the depth image is obtained, and the three-dimensional coordinates of the rip current feature points are obtained according to the camera model and the parameters obtained by the calibration camera;

Step 3.5: Take the position information of the midpoint of the rip current as the position of the rip current, and divide the dangerous area at the left and right edge points of the obtained rip current. Hazardous areas, for example: add 5m to the left of the left edge point and 5m to the right of the right edge point.

Step 4: Based on the location information of the feature points of the rip current, divide the dangerous area and feed it back to the relevant staff, who will remind tourists by marking.

Of course, the above description is not intended to limit the present invention, and the present invention is not limited to the above examples. Changes, modifications, additions or substitutions made by those skilled in the art within the essential scope of the present invention should also belong to the present invention. the scope of protection of the invention.

Claims (3)

1. a rip current detection method based on binocular camera, is characterized in that: comprise the steps: Step 1: Collect the image set for training the convolutional neural network, and train the convolutional neural network; The specific steps for training the convolutional neural network in step 1 are as follows: Step 1.1: Image data preprocessing; specifically includes the following steps: Step 1.1.1: Process the training data collected in step 1, and use a set of images collected by a single-sided camera for processing; convert the collected training data into a data format that TensorFlow can recognize; Step 1.1.2: Add a label according to the image, put the image and label into the array, and convert the array into a format that Tensorflow can recognize; Step 1.1.3: Normalize the image including cropping and supplementing; Step 1.2: Build a convolutional neural network model based on the Tensorflow framework; The classic convolutional neural network LeNet-5 model is adopted; the model is divided into 7 layers: convolutional layer - pooling layer - convolutional layer - pooling layer - fully connected layer - fully connected layer - fully connected output layer; among them, The convolutional layer extracts the preliminary offshore current features, the pooling layer extracts the main features of the offshore current, and the fully connected layer summarizes the features of each part; Step 1.3: Use the constructed convolutional neural network model to train the neural network for identifying rip currents; Step 2: Use the image recognition algorithm of the convolutional neural network to identify the rip current on the collected image; determine whether the rip current exists; If: the judgment result is that the rip current exists, then find the feature points of the rip current, and extract the feature points of the rip current; the feature points include the midpoint and the left and right edge points of the rip current; Or the judgment result is that the rip current does not exist, then the image is collected again for processing; Step 3: Use the binocular positioning algorithm to locate the identified feature points of the rip current, and use the position of the feature points of the rip current to identify the position of the rip current; Obtaining the three-dimensional coordinates of the rip current based on binocular stereo vision technology; Specifically include the following steps: Step 3.1: Calibrate the left and right cameras; The camera is calibrated by Zhang Zhengyou's calibration method, and the internal and external parameters of the left and right cameras are obtained; Step 3.2: Correct the left and right camera images; Use the internal and external parameters obtained by the calibrated left and right cameras to perform distortion correction and stereo correction on the image; Step 3.3: Stereo matching the image; Use the SGBM algorithm to perform stereo matching on the image, and finally obtain the disparity map; Step 3.4: Obtain the three-dimensional coordinate information of the rip current; Using the disparity map and the internal parameters of the left and right cameras, the depth image is obtained, and the three-dimensional coordinates of the rip current feature points are obtained according to the camera model and the parameters obtained by the calibration camera; Step 3.5: Take the position information of the midpoint of the rip current as the position of the rip current, and divide the dangerous area at the left and right edge points of the obtained rip current, that is, according to the nature of the rip current in the area of the rip current, according to the actual situation Expand to get dangerous areas; Step 4: Based on the location information of the feature points of the rip current, divide the dangerous area and feed it back to the relevant staff, who will remind tourists by marking. 2. The method for detecting rip current based on a binocular camera according to claim 1, wherein in step 2, the binocular camera includes a left camera and a right camera, and since the data collected by the left and right cameras is basically the same, a single camera is adopted. Side camera images for rip current identification. 3. the rip current detection method based on binocular camera according to claim 1, is characterized in that: in step 3.3, SGBM is a kind of semi-global block matching algorithm, and SGBM algorithm specifically comprises the steps: S1: image preprocessing; Use the horizontal Sobel operator to process the image, map the pixels to obtain a new image, and preprocess the gradient information of the original image; S2: cost calculation; The cost calculation is divided into two steps: one is the gradient cost calculation of the gradient information of the preprocessed image obtained by the sampling-based method; the second is the SAD cost calculation of the original image obtained by the sampling-based method; S3: dynamic programming; Accumulate energy in each direction according to the idea of dynamic programming, and then add the matching cost in each direction to get the total matching cost; S4: post-processing; The post-processing part needs to perform uniqueness detection, sub-pixel interpolation and left-right consistency detection.

Images