CN114821778A - A method and device for dynamic recognition of underwater fish body posture - Google Patents

Abstract

The invention provides a method and a device for dynamically identifying underwater fish posture, wherein the method comprises the following steps: determining a fish body image to be recognized; and performing target detection on the spliced image based on the fish body posture recognition model to obtain a target detection frame image, then performing down-sampling on the target detection frame image, generating a feature pyramid based on the down-sampled target detection frame image, and determining the fish body posture based on the features in the feature pyramid. The invention can determine the posture of the fish body in the moving state with high precision.

Description

A method and device for dynamic recognition of underwater fish body posture

technical field

The invention relates to the technical field of image recognition, in particular to a method and device for dynamic recognition of underwater fish body posture.

Background technique

Aquaculture is the use of the waters available for aquaculture, in accordance with the ecological habits of the aquaculture objects and the requirements for the environmental conditions of the waters, using aquaculture technology and facilities to engage in aquatic economic aquaculture. The behavior of fish swimming is closely related to the environment in which it is located. Detecting and recognizing the behavior and posture of fish can help to judge its health.

At present, traditional image processing techniques (such as binarization, shape fitting, gray value detection) and convolutional neural networks are mostly used for fish gesture recognition, but the above methods can only perform gesture recognition for fish in a stationary state. It is impossible to accurately recognize the posture of the fish in the moving state.

SUMMARY OF THE INVENTION

The present invention provides a method and a device for dynamically recognizing the posture of an underwater fish body, which are used to solve the defect that the posture recognition accuracy of the fish in a moving state is low in the prior art.

The invention provides a method for dynamic recognition of underwater fish body posture, comprising:

Determine the image of the fish to be identified;

Based on the fish pose recognition model, target detection is performed on the spliced image, and after the target detection frame image is obtained, the target detection frame image is downsampled, and a feature pyramid is generated based on the downsampled target detection frame image, and based on the The features in the feature pyramid determine the fish body posture; the spliced image is obtained after splicing the data-enhanced fish body images to be identified;

The fish body gesture recognition model is obtained by training based on the sample fish body image and the sample fish body gesture label.

According to a method for dynamic recognition of underwater fish body posture provided by the present invention, the target detection is performed on a spliced image based on a fish body posture recognition model, and after a target detection frame image is obtained, the target detection frame image is down-sampled, And generate a feature pyramid based on the down-sampled target detection frame image, and determine the fish body posture based on the features in the feature pyramid, including:

Based on the input layer of the fish body gesture recognition model, data enhancement is performed on the fish body image to be recognized, and multiple images obtained after the data enhancement are spliced to obtain the spliced image, and the spliced image is processed target detection to obtain the target detection frame image;

Based on the sampling layer of the fish pose recognition model, slice and convolute the target detection frame image in sequence to obtain an initial feature map, and perform multiple downsampling on the initial feature map to obtain a downsampled feature map ;

Based on the pyramid layer of the fish body gesture recognition model, the feature map of the upper layer is upsampled and then fused with the feature map of the next layer to obtain a first feature pyramid, and the feature map of the next layer is downsampled. Fusion with the feature map of the previous layer to obtain a second feature pyramid, and the feature map of the first layer is the down-sampling feature map;

Based on the fish body posture being the prediction layer of the recognition model, the fish body posture is determined according to the features in the first feature pyramid and the second feature pyramid.

According to the method for dynamically recognizing the posture of an underwater fish body provided by the present invention, before performing target detection on the spliced image, the method further includes: scaling the spliced image to a preset standard size.

According to a method for dynamic recognition of underwater fish body posture provided by the present invention, the initial feature map is down-sampled multiple times to obtain a down-sampled feature map, including:

Downsampling the current feature map to obtain the current initial downsampling feature map, and fusing the previous downsampling feature map and the current initial downsampling feature map based on the residual component of the sampling layer to obtain the current downsampling feature map ; the first current feature map is the initial feature map.

According to a method for dynamic recognition of underwater fish body posture provided by the present invention, the obtained down-sampling feature map further includes:

Convolution processing, normalization processing and maximum pooling processing are sequentially performed on the down-sampled feature map.

According to a method for dynamic recognition of underwater fish body posture provided by the present invention, the determination of the fish body posture based on the features in the feature pyramid includes:

Determine a plurality of candidate detection frames based on the features in the feature pyramid;

Based on the non-maximum value suppression algorithm, a fish body detection frame is obtained from a plurality of candidate detection frames, and gesture recognition is performed based on the fish body detection frame to determine the fish body posture.

According to a method for dynamic recognition of underwater fish body posture provided by the present invention, the loss value of the fish body posture recognition model is determined based on the following formula:

Loss=-1/n∑(t[i]log(o[i])+(1-t[i])log(1-o[i]));

Among them, Loss represents the loss value of the fish body posture recognition model, o[i] represents the sample predicted fish body posture obtained by the fish body posture recognition model predicting the sample fish body image, and t[i] represents the sample fish body posture Fish pose label.

The present invention also provides an underwater fish posture dynamic recognition device, comprising:

a determining unit for determining the image of the fish to be identified;

The recognition unit is used for performing target detection on the spliced image based on the fish body gesture recognition model, after obtaining the target detection frame image, down-sampling the target detection frame image, and generating a feature pyramid based on the down-sampled target detection frame image , and determine the fish body posture based on the features in the feature pyramid; the stitched image is obtained by stitching the data-enhanced fish body images to be identified;

The fish body gesture recognition model is obtained by training based on the sample fish body image and the sample fish body gesture label.

The present invention also provides an electronic device, comprising a memory, a processor and a computer program stored in the memory and running on the processor, when the processor executes the program, the underwater fish as described above can be realized Body pose dynamic recognition method.

The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, implements any one of the above-mentioned methods for dynamically recognizing the posture of an underwater fish.

The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements any one of the above-mentioned methods for dynamically recognizing the posture of an underwater fish.

The underwater fish body posture dynamic recognition method and device provided by the present invention, based on the fish body posture recognition model, perform target detection on the spliced image, and after obtaining the target detection frame image, the target detection frame image is down-sampled, so that after the down-sampling The target detection frame image not only retains the feature information of the image, but also reduces the calculation parameters of the model and improves the recognition efficiency of the model. In addition, a feature pyramid is generated based on the down-sampled target detection frame image, so that the fish pose can be accurately determined based on the features of each scale in the feature pyramid effectively.

Description of drawings

In order to explain the present invention or the technical solutions in the prior art more clearly, the following will briefly introduce the accompanying drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are the For some embodiments of the invention, for those of ordinary skill in the art, other drawings can also be obtained according to these drawings without any creative effort.

Fig. 1 is one of the schematic flow sheets of the underwater fish body posture dynamic identification method provided by the present invention;

Fig. 2 is the structural representation of the sampling layer of the fish body gesture recognition model provided by the present invention;

Fig. 3 is the structural representation of Focus in the sampling layer provided by the present invention;

4 is a schematic structural diagram of CSP1_X in the sampling layer provided by the present invention;

5 is a schematic structural diagram of a convolutional layer CBL provided by the present invention;

Fig. 6 is the structural representation of CSP2_X in the pyramid layer provided by the present invention;

7 is a schematic structural diagram of a residual component provided by the present invention;

Fig. 8 is the structural representation of SPP provided by the present invention;

9 is a schematic diagram of a model performance index curve provided by the present invention;

Fig. 10 is the second schematic flow chart of the underwater fish posture dynamic identification method provided by the present invention;

11 is a schematic structural diagram of an underwater fish body posture dynamic recognition device provided by the present invention;

FIG. 12 is a schematic structural diagram of an electronic device provided by the present invention.

Detailed ways

In order to make the objectives, technical solutions and advantages of the present invention clearer, the technical solutions in the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention. , not all examples. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

At present, traditional image processing techniques (such as binarization, shape fitting, gray value detection) and convolutional neural networks are mostly used for fish gesture recognition, but the above methods can only perform gesture recognition for fish in a stationary state. It is impossible to accurately recognize the posture of the fish in the moving state.

In addition, there is also a fish body gesture recognition through support vector machine technology, but this method relies on complex algorithms such as extracting the following feature vectors and sliding windows, which is inefficient, and this method requires additional hardware facilities and high maintenance costs.

In this regard, the present invention provides a dynamic recognition method for underwater fish body posture. Fig. 1 is one of the schematic flow charts of the underwater fish posture dynamic recognition method provided by the present invention, as shown in Fig. 1, the method comprises the following steps:

Step 110: Determine the image of the fish body to be identified.

Here, the fish body image to be recognized is the image for which fish body gesture recognition needs to be performed. The image of the fish body to be identified may be acquired by an image acquisition device, where the image acquisition device may be a camera or a camera, or a smart device such as a smartphone, tablet, or computer equipped with a camera.

Step 120: Perform target detection on the spliced image based on the fish pose recognition model, after obtaining the target detection frame image, downsample the target detection frame image, and generate a feature pyramid based on the downsampled target detection frame image, and based on the feature The features in the pyramid determine the posture of the fish body; the stitched image is obtained by stitching the image of the fish body to be identified after data enhancement;

The fish pose recognition model is trained based on sample fish images and their sample fish pose labels.

Specifically, the stitched image is obtained by stitching the data-enhanced fish images to be identified. For example, after determining the fish body image to be recognized, data enhancement can be performed on it to obtain multiple enhanced images. Among them, these enhanced images may be obtained by randomly scaling, randomly cropping, and randomly arranging the fish images to be identified based on the Mosaic data enhancement algorithm. After data enhancement is performed on the image of the fish to be recognized, the acquired stitched image contains the attitude information of different angles and sizes of the fish to be recognized, so that more detailed attitude information of the fish to be recognized can be obtained based on the stitched image when it moves.

After the stitched image is obtained, target detection is performed on the stitched image, so that the target object that needs gesture recognition can be accurately extracted from the stitched image, and the target detection frame image is obtained. After obtaining the target detection frame image, down-sampling it can not only not lose the feature information of the image, but also reduce the calculation parameters of the model and improve the recognition efficiency of the model.

After down-sampling the target detection frame image, a feature pyramid is generated based on the down-sampled target detection frame image, and the fish pose is determined based on the features in the feature pyramid. Among them, the feature pyramid can be the top-down FPN (feature pyramid networks) feature pyramid and the bottom-up PAN (Pyramid Attention Network) feature pyramid. The combination of the two increases the bottom-up feature based on the FPN feature pyramid. The enhancement in direction enables the top-level features to obtain the position information of the bottom-level features, thereby improving the gesture recognition accuracy.

It should be understood that, in this embodiment of the present invention, a fish body gesture recognition model can be obtained by pre-training, which can be specifically implemented by performing the following steps: first, collect a large number of sample fish body images, and determine their corresponding sample fish body gesture labels through manual annotation. Immediately, the initial model is trained based on the sample fish images and their sample fish pose labels, thereby obtaining a fish pose recognition model.

As an optional embodiment, the image of the sample fish body comes from a fish pond with a diameter of 5 meters and a depth of 80 cm. The water in the fish pond comes from filtered seawater, the water temperature is heated to 15 degrees, and the camera is placed in the pond At the bottom, use the video recording method to obtain the sample fish image video, the video pixel is 1920*1080, 25 frames per second, and the format is mp4. The video was recorded for 1 hour, and Python was used to intercept the video obtained from the recording, and a series of initial sample fish images in jpg format were obtained by taking one image every two seconds. A total of 1800 initial sample fish images were obtained and Saved to hard disk, and then deleted some low-quality initial sample fish images, such as missing fish, a lot of overlapping fish, no fish, and images with blurred fish, and finally got 800 sample fish images.

In addition, after the sample fish image is obtained, the image enhancement algorithm ACE (Automatic Color Equalization) can be used to process the sample fish image. The ACE algorithm is derived from the retinex algorithm, which can adjust and stretch the contrast of the image to achieve color and saturation. standardization. Correct the pixel value by differentially calculating the relative light-dark relationship between the target point and the surrounding pixels, which has a good enhancement effect, can correct the color cast, realize the dehazing effect, noise reduction effect, and can also realize the brightness enhancement of the dark scene. The sample fish image of the original image does not have the blue-green color cast of the original image, the brightness has been improved, and the underwater blur has also been improved. In addition, makesense can be used to label the sample fish image, and different labels are labeled for fish bodies with different postures, and overlapping and missing fish are not labeled, and the sample fish body posture labels are obtained. Then use python to divide the sample fish image, for example, it can be randomly divided into training set and test set according to the ratio of 8:2.

In the method for dynamic recognition of underwater fish body posture provided by the embodiment of the present invention, based on the fish body posture recognition model, target detection is performed on the spliced image, and after the target detection frame image is obtained, the target detection frame image is down-sampled, so that after the down-sampling, the target detection frame image is down-sampled. The target detection frame image not only retains the feature information of the image, but also reduces the calculation parameters of the model and improves the recognition efficiency of the model. In addition, a feature pyramid is generated based on the down-sampled target detection frame image, so that the fish pose can be accurately determined based on the features of each scale in the feature pyramid effectively.

Based on the above embodiment, based on the fish body gesture recognition model, target detection is performed on the spliced image, after obtaining the target detection frame image, the target detection frame image is down-sampled, and a feature pyramid is generated based on the down-sampled target detection frame image, and Determine the fish pose based on the features in the feature pyramid, including:

Based on the input layer of the fish pose recognition model, data enhancement is performed on the fish image to be recognized, and multiple images obtained after data enhancement are spliced to obtain a spliced image, and target detection is performed on the spliced image to obtain a target detection frame image;

Based on the sampling layer of the fish pose recognition model, slicing and convolution operations are performed on the target detection frame image in turn to obtain the initial feature map, and the initial feature map is down-sampled multiple times to obtain the down-sampled feature map;

Based on the pyramid layer of the fish pose recognition model, the feature map of the upper layer is up-sampled and then fused with the feature map of the next layer to obtain the first feature pyramid, and the feature map of the next layer is downsampled and then merged with the feature map of the next layer. The feature maps of one layer are fused to obtain a second feature pyramid, and the feature maps of the first layer are down-sampling feature maps;

The fish body pose is the prediction layer of the recognition model, and the fish body pose is determined according to the features in the first feature pyramid and the second feature pyramid.

Specifically, the fish pose recognition model may adopt the YOLOv5 model structure, which may include a four-layer structure of an input layer, a sampling layer (Backbone layer), a pyramid layer (Neck layer) and a prediction layer (Prediction layer).

Among them, the input layer can include Mosaic data enhancement algorithm, adaptive image scaling algorithm and adaptive anchor box calculation algorithm. The sampling layer can contain focus structure, CSP1_X structure and SPP structure. Pyramid layers can contain CSP2_X, FPN and PAN structures. The prediction layer can contain GIOU_Loss and NMS non-maximum suppression algorithm.

Based on the input layer of the fish pose recognition model, data enhancement is performed on the fish body image to be recognized, and multiple images obtained after data enhancement are spliced to obtain a spliced image, and target detection is performed on the spliced image to obtain a target detection frame image. Among them, Mosaic data enhancement algorithm can be used for data enhancement of fish images to be recognized, such as random scaling, random cropping, random arrangement and splicing of any four images, which improves the detection accuracy of small targets and occluded targets. It reduces the memory requirements of the model and enriches the dataset. Mosaic9 is based on Mosaic, which increases the number of randomly scaled, randomly cropped, and randomly arranged pictures to 9, which further improves the robustness and reduces the model's memory requirements. In the YOLO algorithm, for different data sets, there will be anchor boxes with initial length and width. In the process of training the fish pose recognition model, the predicted frame is output based on the initial anchor frame, and then compared with the real frame, the difference between the two is calculated, and then the network parameters are updated in the reverse direction. At each training time, the optimal anchor box values in different training sets are adaptively calculated.

As shown in Figure 2, the sampling layer contains focus structure, CSP1_X structure and SPP structure. Focus structure, by inputting the original 608 × 608 × 3 image into the Focus structure, after the slicing operation, it becomes a 304 × 304 × 12 feature map, and after a convolution operation of 32 convolution kernels, it finally becomes 304 × 304 ×32 feature map, the function of this structure is to avoid the loss of image feature information in the process of image downsampling, so that more sufficient features can be extracted when using convolution operations. This module performs slicing operation before backbone, as shown in Figure 3, selects the value of every other pixel for downsampling, obtains four pictures with complementary features without losing information, and concentrates the W and H information into the channel space, the output space has been expanded by four times, and the original three channels have become twelve channels, so that after the convolution operation, a double downsampling feature map without information loss can be obtained. The Focus structure reduces the cost of convolution and improves the speed. . In addition, as shown in Figure 4 and Figure 5, a convolution layer is set before the CSP1_X structure, the size of the convolution kernel is 3 × 3, and the stride is 2, so that the size of the feature map is reduced by half, and the downsampling is achieved. Function, there are 4 CSP1_X structures in Backbone, and the final output feature map size is 19×19.

Based on the pyramid layer of the fish pose recognition model, the feature map of the upper layer is up-sampled and then fused with the feature map of the next layer to obtain the first feature pyramid (ie, the FPN feature pyramid), and the feature map of the next layer is obtained. After downsampling, it is fused with the feature map of the previous layer to obtain the second feature pyramid (ie, the PAN feature pyramid).

For example, the pyramid layer can be an FPN+PAN structure, combining downsampling and upsampling to generate feature pyramids, making the network scale invariant. The feature pyramid generated by the FPN structure is top-down, and the feature maps extracted by the sampling layer are down-sampled to obtain features with sizes of 76 × 76 × 255, 38 × 38 × 255, and 19 × 19 × 255, where 255 means 255 Dimensional output, FPN fuses the upper-layer feature upsampling result with the next-layer feature to obtain a feature pyramid. The embodiment of the present invention adds a PAN structure on the basis of FPN. Contrary to FPN, the PAN feature pyramid is bottom-up, obtained by fusing the lower-level feature downsampling result of FPN with the upper-level feature. FPN mainly improves the effect of target detection by fusing high and low-level features, especially the detection effect of small-sized targets; PAN structure segments the network shallow features, and the network shallow feature information is very important for target detection, because target detection is pixel-level The shallow information is mostly edge and other features. The combination of the two increases the bottom-up enhancement on the basis of FPN, so that the top-level features can obtain the position information of the bottom-level features, which improves the detection effect of large objects. Among them, the pyramid layer also includes the CSP2_X structure shown in FIG. 6 .

After the first pyramid and the second pyramid are obtained, the fish body pose is identified and determined by the prediction layer of the recognition model according to the features in the first feature pyramid and the second feature pyramid.

Based on any of the foregoing embodiments, performing target detection on the stitched image further includes: scaling the stitched image to a preset standard size.

Specifically, the input layer may also include an adaptive image scaling algorithm, which is used to scale the stitched image to a preset standard size before performing target detection on the stitched image, so as to add as few black borders as possible, thereby greatly improving the model's performance. Calculate speed.

Based on any of the above embodiments, the initial feature map is down-sampled multiple times to obtain a down-sampled feature map, including:

Downsampling the current feature map to obtain the current initial downsampling feature map, and fuses the previous downsampling feature map and the current initial downsampling feature map based on the residual component of the sampling layer to obtain the current downsampling feature map; the first The current feature map is the initial feature map.

As shown in Figure 7, the sampling increase also includes multiple residual components and a tensor splicing module, and uses SiLU as the activation function. The residual component uses the identity mapping to perform the previous downsampling feature map and the current initial downsampling feature map Fusion to obtain the current down-sampled feature map to prevent gradient explosion and network degradation during the convolution process, thereby enhancing the learning ability of the model, reducing computational bottlenecks, and reducing memory consumption.

Based on any of the above-mentioned embodiments, a down-sampling feature map is obtained, which further includes:

Convolution, normalization, and max pooling are performed on the downsampled feature maps in sequence.

Specifically, the sampling layer may also include an SPP structure. As shown in Figure 8, the SPP structure is first processed by 1×1 convolution and then normalized. The main operation steps of normalization are to calculate the mean and variance of all data, and then The difference between the pixel value and the mean is divided by the variance for normalization, and an offset factor and a scale change factor are added to control the normalized value. The value of the factor is learned by the model during training. Then, 5/9/13 can be used for maximum pooling, concat tensor splicing, and finally SiLU is used as the activation function, which effectively avoids image distortion caused by cropping and scaling operations on the image area. The receptive field solves the problem of repeated extraction of image features by the model, greatly improves the operation speed and saves the calculation cost.

Based on any of the above-mentioned embodiments, the fish body posture is determined based on the features in the feature pyramid, including:

Determine multiple candidate detection frames based on the features in the feature pyramid;

Based on the non-maximum suppression algorithm, a fish body detection frame is obtained by screening multiple candidate detection frames, and gesture recognition is performed based on the fish body detection frame to determine the fish body posture.

Specifically, based on the features in the feature pyramid, multiple candidate detection frames can be obtained. Usually, these candidate detection frames contain the same content. Therefore, it is necessary to filter out the best detection frame as the fish body detection frame for gesture recognition. Test results can be simplified. Therefore, the embodiment of the present invention is based on the non-maximum suppression algorithm (NMS), and the fish detection frame is obtained from multiple candidate detection frames, so that some occlusion and overlapping targets can be improved to a certain extent, and the original detection frame is not detected. The target can be detected and identified after the NMS operation.

Among them, when the non-maximum suppression operation is performed, the corresponding loss value can be determined: GIoUloss=1-(IoU-|Ac-U|)/|Ac|, where IoU is the intersection ratio, Ac is the prediction frame and The area of the minimum closure of the ground truth box, U is the union area of the predicted box and the ground truth box. The meaning of this formula is: first calculate the minimum closure area of the two boxes, then calculate the proportion of the IoU and the area that does not belong to the two boxes in the closure area to the closure area, and finally subtract this proportion from the IoU to get the GIoU , the final Boundingbox loss is 1-GIoU.

Based on any of the above embodiments, the loss value of the fish body gesture recognition model is determined based on the following formula:

Loss=-1/n∑(t[i]log(o[i])+(1-t[i])log(1-o[i]));

Among them, Loss represents the loss value of the fish pose recognition model, o[i] represents the sample predicted fish pose obtained by the fish pose recognition model predicting the sample fish image, and t[i] represents the sample fish pose label.

In addition, the SiLU activation function used in the embodiment of the present invention has the characteristics of no upper bound and lower bound, smooth and non-monotonic, and is a special case of the Swish activation function: f(x)=x*σ(x)f′(x )=f(x)+σ(x)(1−f(x)).

As an optional embodiment, the operating environment of the fish body gesture recognition model can use the colab platform of google, the data and YOLOv5 model files are stored on the google cloud disk, and the model is imported into the colab platform to run during training. YOLOv5 uses version 6.1 and python version For 3.7, the required library files are installed using the requirements file, the cloud disk capacity is 15GB, the memory capacity is 8GB, the GPU is Tesla K80, the video memory size is 11441MB, and CUDA is enabled during training.

The parameter settings of the YOLOv5 model are: use the yolov5n model, use the hyperparameters optimized for scratch-low, set the epoch to 100, set the batch-size to 16, set the image size to 640, enable cache, enable agnostic-nms, and enable augment , turn on FP16 half, set optimizer to AdamW, turn on label-smoothing and set it to 0.1, set conf-thres to 0.5, and set iou-thres to 0.5.

After Cache is turned on, the dataset will be imported into the video memory before training, and there is no need to read it from the hard disk every time it is used, which speeds up the training speed. agnostic-nms is nms non-maximum suppression, which can reduce repeated and redundant target boxes after enabling it, iou-thres is the intersection ratio used when non-maximum suppression is enabled, conf-thres is the confidence threshold setting, Only target boxes with a confidence greater than this value are displayed in the training results. Augment is a data enhancement method based on the HSV color space. After opening, it can expand the data volume to a certain extent and improve the robustness of the model. AdamW is a gradient descent algorithm obtained after adding L2 regularization on the basis of Adam, which reduces the overfitting phenomenon and has a significantly better convergence speed than Adam and SGD. FP16 half adopts half floating-point precision, which can greatly reduce the memory usage and speed up the operation without affecting the accuracy of the model. Label-smoothing is a regularization strategy. It mainly uses soft one-hot to add noise, which reduces the weight of the real sample label category when calculating the loss function, and finally has the effect of suppressing overfitting and improving the generalization of the model. transformation ability.

In addition, in order to improve the convergence speed of the model, improve the training efficiency, and reduce the number of iterations, the hyperparameters of the YOLOv5 model are obtained using the evolve hyperparameter evolution algorithm. Each epoch is trained until the 10th generation.

The initial value of the hyperparameter is the value in the hyp.scratch-low file in YOLOv5, and the fitness of the hyperparameter evaluation index is set to 0.1×mAP(0.5)+0.9×mAP(0.5:0.95). This method obtains the optimal value by maximizing the fitness index. The hyperparameter of the method uses a genetic algorithm. For each generation in the evolution process, the previous generation with the highest fitness score is selected for mutation. All hyperparameters will mutate to other values about 20% of the time.

The optimized values are: lr0 learning rate 0.008, lr1 learning rate 0.07, learning momentum 0.85, weight decay 0.0006, warm-up algebra 3.4, warm-up learning momentum 0.95, warm-up learning initial bias learning rate 0.09, anchor box size 0.065, iou Threshold 0.2, Hue Shift 0.001, Saturation Boost 0.63, Brightness Boost 0.43, Image Rotation Prob 0.0009, Image Shift Prob 0.12, Image Scaling Prob 0.9, Image Flip Prob 0.001, Image Mirror Prob 0.5, Mosaic Image Enhance Prob 0.85, Mixup Image Mix The probability is 0.05, and the number of anchor boxes in each layer is 4.6.

Then, use the computer equipped with the deployed YOLOv5 6.1 version to train the image. The model has its own image enhancement algorithm, which can realize the mirror flip, crop and stitch, rotate and translate, copy and mix, adjust the brightness, hue and saturation of the original image. Expand the dataset. The YOLOv5 algorithm iteratively updates the weight matrix and bias in the forward propagation through gradient descent to reduce the loss between the predicted frame and the real frame, and obtains the value of the error cost function according to the updated weights, and uses the optimizer to adjust Learning rate, and finally get the fish pose recognition model.

Among them, the following parameters can be used to evaluate the performance of the model:

FN: It is judged as a negative sample, but it is actually a positive sample.

FP: It is judged as a positive sample, but it is actually a negative sample.

TN: It is judged as a negative sample, which is actually a negative sample.

TP: It is judged as a positive sample, which is actually a proof sample.

The intersection and union ratio IoU=A∩B/A∪B.

Precision precision=TP/(TP+FP).

Recall rate recall=TP/(TP+FN).

F1-score=2(Precision×Recall)/(Precision+Recall).

mAP=1/m∑AP(i), i∈[0,m), i∈N+, m is the number of categories, AP(i) is the average precision of the i-th category.

mAP(0.5) represents mAP when IoU is 0.5, TP is the number of detection boxes with IoU>0.5, and FP is the number of detection boxes with IoU<=0.5.

mAP (0.5:0.95) represents the average mAP over different IoU thresholds (from 0.5 to 0.95, step size 0.05).

YOLOv5 is divided into five structures according to depth and width, namely v5n, v5s, v5m, v5l, and v5x. In this embodiment of the present invention, the v5n structure with the simplest structure can be adopted. This structure has the fastest running speed, compared with the other four models. , the prediction accuracy is not significantly reduced.

As shown in Figure 9, the fish pose recognition model achieves 95.1% precision, 94.9% recall, 97.8% mAP (0.5) score and 83.0% mAP (0.5:0.95) score, each evaluation index is in 100 All epochs have basically converged. Through the evaluation of the mAP score of the model, it is further proved that the method provided by the embodiment of the present invention has good robustness and accuracy, and the recognition speed can be as high as 120FPS, which can realize the quasi-real-time moving target of the fish body in the underwater environment Fully automatic identification and positioning.

Based on any of the above embodiments, the present invention also provides a method for dynamic recognition of underwater fish body posture, as shown in FIG. 10 , the method includes:

First, video image data is acquired, and image preprocessing and image annotation are sequentially performed on the video image data. Next, the labeled video image data is randomly divided into training set and test set, and the training set is used to build a model and train a fish pose detection model (the model can be the YOLOv5-F model).

After the model training is completed, use the test set to evaluate the performance and accuracy of the model to determine whether it meets the standard. If so, the gesture recognition can be performed on the real-time video data;

The device for dynamic recognition of underwater fish body posture provided by the present invention is described below. The device for dynamic recognition of underwater fish body posture described below and the method for dynamic recognition of underwater fish body posture described above can be referred to each other correspondingly.

Based on any of the above embodiments, the present invention also provides an underwater fish body posture dynamic recognition device, as shown in FIG. 11 , the device includes:

a determining unit 1110, configured to determine the image of the fish to be identified;

The identification unit 1120 is configured to perform target detection on the spliced image based on the fish body gesture recognition model, after obtaining the target detection frame image, downsample the target detection frame image, and generate features based on the downsampled target detection frame image Pyramid, and determine the fish body posture based on the features in the feature pyramid; the spliced image is obtained by splicing the data-enhanced fish body images to be identified;

The fish body gesture recognition model is obtained by training based on the sample fish body image and the sample fish body gesture label.

Based on any of the above embodiments, the identifying unit 1120 includes:

The input unit is used for performing data enhancement on the fish body image to be recognized based on the input layer of the fish body gesture recognition model, and splicing the multiple images obtained after the data enhancement to obtain the spliced image, and The stitched image is subjected to target detection to obtain the target detection frame image;

The sampling unit is used for sequentially slicing and convolution operations on the target detection frame image based on the sampling layer of the fish body gesture recognition model to obtain an initial feature map, and performing multiple downsampling on the initial feature map, Get the down-sampling feature map;

The pyramid unit is used to upsample the feature map of the upper layer and fuse the feature map of the next layer to obtain the first feature pyramid based on the pyramid layer of the fish body gesture recognition model, and to obtain the first feature pyramid, and to the features of the next layer. After the image is down-sampled, it is fused with the feature map of the previous layer to obtain a second feature pyramid, and the feature map of the first layer is the down-sampling feature map;

A prediction unit, configured to determine the fish body posture according to the features in the first feature pyramid and the second feature pyramid based on the fish body posture being a prediction layer of a recognition model.

Based on any of the foregoing embodiments, the apparatus further includes a scaling unit, configured to scale the stitched image to a preset standard size before performing target detection on the stitched image.

Based on any of the foregoing embodiments, the sampling unit is configured to:

Downsampling the current feature map to obtain the current initial downsampling feature map, and fusing the previous downsampling feature map and the current initial downsampling feature map based on the residual component of the sampling layer to obtain the current downsampling feature map ; the first current feature map is the initial feature map.

Based on any of the foregoing embodiments, the apparatus further includes:

The processing unit is configured to sequentially perform convolution processing, normalization processing and maximum pooling processing on the down-sampling feature map after obtaining the down-sampling feature map.

Based on any of the above embodiments, the identifying unit 1120 includes:

a candidate unit for determining a plurality of candidate detection frames based on the features in the feature pyramid;

The screening unit is configured to screen out a plurality of candidate detection frames to obtain a fish body detection frame based on a non-maximum value suppression algorithm, and perform gesture recognition based on the fish body detection frame to determine the fish body posture.

Based on any of the above embodiments, the loss value of the fish body gesture recognition model is determined based on the following formula:

Loss=-1/n∑(t[i]log(o[i])+(1-t[i])log(1-o[i]));

Among them, Loss represents the loss value of the fish body posture recognition model, o[i] represents the sample predicted fish body posture obtained by the fish body posture recognition model predicting the sample fish body image, and t[i] represents the sample fish body posture Fish pose label.

FIG. 12 is a schematic structural diagram of an electronic device provided by the present invention. As shown in FIG. 12 , the electronic device may include: a processor (processor) 1210, a memory (memory) 1220, a communication interface (Communications Interface) 1230 and a communication bus 1240, The processor 1210 , the memory 1220 , and the communication interface 1230 communicate with each other through the communication bus 1240 . The processor 1210 can call the logic instructions in the memory 1220 to execute a method for dynamic recognition of underwater fish body posture. The method includes: determining a fish body image to be recognized; After the frame image is detected, the target detection frame image is downsampled, and a feature pyramid is generated based on the downsampled target detection frame image, and the fish pose is determined based on the features in the feature pyramid; The fish body image to be identified after data enhancement is obtained by splicing; the fish body gesture recognition model is obtained by training based on the sample fish body image and the sample fish body gesture label.

In addition, the above-mentioned logic instructions in the memory 1220 can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present invention can be embodied in the form of a software product in essence, or the part that contributes to the prior art or the part of the technical solution. The computer software product is stored in a storage medium, including Several instructions are used to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, Read-Only Memory (ROM, Read-Only Memory), Random Access Memory (RAM, Random Access Memory), magnetic disk or optical disk and other media that can store program codes .

In another aspect, the present invention also provides a computer program product, the computer program product comprising a computer program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions, when the program instructions are executed by a computer When executing, the computer can execute the underwater fish body posture dynamic recognition method provided by the above methods, and the method includes: determining the fish body image to be recognized; based on the fish body posture recognition model, performing target detection on the spliced images to obtain a target detection frame After the image, the target detection frame image is downsampled, and a feature pyramid is generated based on the downsampled target detection frame image, and the fish body posture is determined based on the features in the feature pyramid; the stitched image is a data enhancement method. The fish body image to be recognized is obtained after splicing; the fish body gesture recognition model is obtained by training based on the sample fish body image and the sample fish body gesture label.

In yet another aspect, the present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, it is realized to execute the above-mentioned methods for dynamic recognition of underwater fish body posture, The method includes: determining a fish body image to be recognized; performing target detection on a spliced image based on a fish body gesture recognition model, and after obtaining a target detection frame image, down-sampling the target detection frame image, and based on the down-sampled target The detection frame image generates a feature pyramid, and the fish body posture is determined based on the features in the feature pyramid; the spliced image is obtained by splicing the data-enhanced fish body images to be identified; the fish body posture recognition model is based on The sample fish images and their sample fish pose labels are trained.

The device embodiments described above are only illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in One place, or it can be distributed over multiple network elements. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution in this embodiment. Those of ordinary skill in the art can understand and implement it without any creative effort.

From the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and certainly can also be implemented by hardware. Based on this understanding, the above-mentioned technical solutions can be embodied in the form of software products in essence or the parts that make contributions to the prior art, and the computer software products can be stored in computer-readable storage media, such as ROM/RAM, magnetic A disc, an optical disc, etc., includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to perform the methods described in various embodiments or some parts of the embodiments.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, but not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that it can still be The technical solutions described in the foregoing embodiments are modified, or some technical features thereof are equivalently replaced; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims (10)

1. a method for dynamic identification of underwater fish body posture, is characterized in that, comprises: Determine the image of the fish to be identified; Based on the fish pose recognition model, target detection is performed on the spliced image, and after the target detection frame image is obtained, the target detection frame image is downsampled, and a feature pyramid is generated based on the downsampled target detection frame image, and based on the The features in the feature pyramid determine the fish body posture; the stitched image is obtained by stitching the data-enhanced fish body images to be identified; The fish body gesture recognition model is obtained by training based on the sample fish body image and the sample fish body gesture label. 2. underwater fish body posture dynamic recognition method according to claim 1, is characterized in that, described based on fish body posture recognition model, carries out target detection to spliced image, after obtaining target detection frame image, to described target detection The frame image is down-sampled, and a feature pyramid is generated based on the down-sampled target detection frame image, and the fish body pose is determined based on the features in the feature pyramid, including: Based on the input layer of the fish body gesture recognition model, data enhancement is performed on the fish body image to be recognized, and multiple images obtained after the data enhancement are spliced to obtain the spliced image, and the spliced image is processed target detection to obtain the target detection frame image; Based on the sampling layer of the fish pose recognition model, slice and convolute the target detection frame image in sequence to obtain an initial feature map, and perform multiple downsampling on the initial feature map to obtain a downsampled feature map ; Based on the pyramid layer of the fish body gesture recognition model, the feature map of the upper layer is upsampled and then fused with the feature map of the next layer to obtain a first feature pyramid, and the feature map of the next layer is downsampled. Fusion with the feature map of the previous layer to obtain a second feature pyramid, and the feature map of the first layer is the down-sampling feature map; Based on the fish body posture being the prediction layer of the recognition model, the fish body posture is determined according to the features in the first feature pyramid and the second feature pyramid. 3 . The method for dynamically recognizing underwater fish body posture according to claim 2 , wherein the performing target detection on the spliced image further comprises: scaling the spliced image to a preset standard size. 4 . 4. The method for dynamic recognition of underwater fish body posture according to claim 2, wherein the initial feature map is subjected to multiple downsampling to obtain a downsampling feature map, comprising: Downsampling the current feature map to obtain the current initial downsampling feature map, and fusing the previous downsampling feature map and the current initial downsampling feature map based on the residual component of the sampling layer to obtain the current downsampling feature map ; the first current feature map is the initial feature map. 5. underwater fish body posture dynamic identification method according to claim 2, is characterized in that, described obtains downsampling feature map, also comprises afterwards: Convolution processing, normalization processing and maximum pooling processing are sequentially performed on the down-sampled feature map. 6. The underwater fish body posture dynamic identification method according to any one of claims 1 to 5, wherein the fish body posture is determined based on the feature in the feature pyramid, comprising: Determine a plurality of candidate detection frames based on the features in the feature pyramid; Based on the non-maximum value suppression algorithm, a fish body detection frame is obtained from a plurality of candidate detection frames, and gesture recognition is performed based on the fish body detection frame to determine the fish body posture. 7. according to the underwater fish body posture dynamic identification method described in any one of claim 1 to 5, it is characterized in that, the loss value of described fish body posture recognition model is determined based on following formula: Loss=-1/n∑(t[i]log(o[i])+(1-t[i])log(1-o[i])); Among them, Loss represents the loss value of the fish body posture recognition model, o[i] represents the sample predicted fish body posture obtained by the fish body posture recognition model predicting the sample fish body image, and t[i] represents the sample fish body posture Fish pose label. 8. an underwater fish body posture dynamic identification device, is characterized in that, comprises: a determining unit for determining the image of the fish to be identified; The recognition unit is used for performing target detection on the spliced image based on the fish body gesture recognition model, after obtaining the target detection frame image, down-sampling the target detection frame image, and generating a feature pyramid based on the down-sampled target detection frame image , and determine the fish body posture based on the features in the feature pyramid; the stitched image is obtained by stitching the data-enhanced fish body images to be identified; The fish body gesture recognition model is obtained by training based on the sample fish body image and the sample fish body gesture label. 9. An electronic device, comprising a memory, a processor and a computer program stored on the memory and running on the processor, wherein the processor implements the program as claimed in claim 1 when executing the program The method for dynamic recognition of underwater fish body posture described in any one of to 7. 10. A non-transitory computer-readable storage medium on which a computer program is stored, wherein the computer program realizes the underwater fish posture as described in any one of claims 1 to 7 when the computer program is executed by the processor Dynamic identification method.

Images