CN105184816A - Visual inspection and water surface target tracking system based on USV and detection tracking method thereof - Google Patents

Abstract

The invention discloses a USV-based visual detection and surface target tracking system, which includes a command center, a surface unmanned boat and a detection system; the command center is the control terminal of the water surface unmanned boat, and communicates with the water surface unmanned boat The surface unmanned boat sends instructions; the surface unmanned boat receives the information sent by the detection system, and completes the task of tracking the surface target through the navigation system; the detection system detects and locates the target object on the water surface to obtain its position coordinates, and sends the information Give the surface unmanned boat while waiting for the next command from the command center; the detection system is integrated with a pan-tilt camera and a millimeter-wave radar. The present invention uses two sensors, the pan-tilt camera and the millimeter-wave radar, to jointly locate the position of the water surface target, making the detection and tracking results more accurate, improving the detection efficiency, greatly saving manpower and financial resources, and having broad market prospects.

Description

USV-based visual detection and surface target tracking system and its detection and tracking method

technical field

The invention relates to unmanned surface craft navigation technology, in particular to a USV-based visual detection and surface target tracking system and a detection and tracking method thereof.

Background technique

Unmanned Surface Vessel (USV) is a recently emerging offshore operation platform, which can complete tasks such as autonomous detection of targets, obstacle avoidance, and search and rescue according to artificially set procedures. At present, under the influence of frequent shipwrecks and the rapid development and practicality of naval equipment, countries pay more and more attention to the monitoring and management of targets in the controlled sea areas, which makes the rapid development of automatic target identification technology on the sea surface. With the increasing frequency of ocean activities and frequent occurrence of shipwrecks, coastal countries pay special attention to the monitoring of maritime activities. The detection and monitoring of maritime targets, especially ships, and accident scene searches after shipwrecks are traditional tasks of coastal countries around the world. Detect target images from visual images such as ships and boats, and further extract a large amount of useful information. It has a wide range of applications for accident scenes, sea bays, port monitoring and marine transportation, fishing supervision, and judging dangers in military warfare. application prospects.

In today's era of rapid development of intelligence, maritime security management and military strategy are gradually becoming more intelligent. As a new type of maritime intelligent body, the unmanned surface vehicle has the characteristics of autonomy, responsiveness, and adaptability. Compared with some air and ground unmanned platforms such as drones, it was developed later. It is mainly affected by two factors. One is The sea environment changes greatly, and the light is greatly affected by the reflection of the water surface and other environmental factors; second, the water surface targets are generally relatively close, and the requirements for the vision system sensor are relatively high. Being able to autonomously and accurately identify and judge targets or obstacles in its operating environment is one of the main tasks of unmanned surface vehicles, so there are high intelligent requirements for its vision and processing systems. Unmanned surface vehicles recognize short-distance targets or obstacles mainly based on the vision system. Therefore, it is of great significance to study the recognition, measurement and tracking of unmanned surface targets.

In the existing technology, for the detection and tracking of moving objects in complex dynamic scenes, the method of assisting manual search is usually used to detect specific moving objects, and the tracking algorithm is used to track the moving objects in the image, which is a kind of semi-autonomous guidance. method, so there are certain limitations.

Contents of the invention

Purpose of the invention: The purpose of the present invention is to solve the deficiencies in the prior art and provide a USV-based visual detection and water surface target tracking system and its detection and tracking method.

Technical solution: A USV-based visual detection and surface target tracking system according to the present invention includes a command center, a surface unmanned vehicle and a detection system; The communication between the human boat and the surface unmanned boat sends instructions; the surface unmanned boat receives the information sent from the detection system, and completes the task of tracking the surface target through the navigation system; Detect and locate its location coordinates, and send the information to the surface unmanned boat, while waiting for the next command from the command center; the detection system integrates a pan-tilt camera and a millimeter-wave radar.

The invention also discloses a detection and tracking method of a USV-based visual detection and water surface target tracking system, which includes the following steps:

(1) process the 2D video image captured by the PTZ camera;

(2) PTZ camera tracking control, adjust the pitch and deflection angle of PTZ camera in real time to keep the target in the center of the image;

(3) Utilize the millimeter-wave radar to further measure the distance between the unmanned surface vehicle and the moving target, and combine the data obtained in step (2) to obtain the 3D data of the moving target to complete the tracking and precise positioning of the moving target;

(4) The navigation system of the unmanned surface craft can autonomously track the moving target on the water surface.

Further, the step (1) includes image sequence preprocessing (including image transformation, image denoising, image enhancement, etc., the purpose of which is to facilitate the subsequent processing of image signals), background modeling and detection of moving objects, moving objects Target segmentation and moving target tracking four steps, the specific process is as follows:

(11) Image sequence preprocessing includes image transformation, image denoising and image enhancement, etc.;

(12) Background modeling and dynamic target detection, that is, moving target detection combined with texture and motion model: using local binary mode to extract texture mode; at the same time, the traditional local binary mode is extended from the spatial domain to the spatio-temporal domain for Extract the motion mode, the specific method is: for extracting the texture mode descriptor For the pixel g _t,c at (x _t,c ,y _t,c ) in the image at time t, consider its eight neighboring pixels g _t,p , p=0,....7, perform binarization and comparison of each neighboring pixel with this pixel, and obtain an eight-bit binary string, that is, a code word LBP ^t (x _t,c , y _t,c ):

${\mathrm{<span\; class="notranslate">\; LBP</span>}}^{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; c</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; c</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}_{\mathrm{<span\; class="notranslate">\; p</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 0</span>}}^{\mathrm{<span\; class="notranslate">\; 7</span>}}\mathrm{<span\; class="notranslate">\; the\; s</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; g</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; p</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; g</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; c</span>}}<span\; class="notranslate">\; )</span>{\mathrm{<span\; class="notranslate">\; 2</span>}}^{\mathrm{<span\; class="notranslate">\; p</span>}}$

in $\mathrm{the\; s}\left(x\right)=\left\{\begin{array}{cc}1,& x\&Greater\; Equal;0\\ 0,& x<0\end{array}\right.$

This codeword characterizes a texture pattern formed by the pixel (x _t,c ,y _t,c ) and its surrounding pixels, and then models each pixel in the scene using the texture pattern and motion model, and then at the classifier level Fusion of background models based on texture mode and motion mode;

(13) Segmentation of moving targets: two background clipping images are generated by using the background modeling method based on the Gaussian mixture model and the embedded feature of the adjacent image block, and then these two background clipping images are combined into one background clipping image; then extract The connected area of the foreground pixels, the connected area and the pixels in its surrounding neighborhood are divided into foreground seed pixels, background seed pixels and unmarked pixels; finally, the connected area containing the moving target and its surrounding area Neighborhoods are segmented more finely;

(14) Tracking of moving targets: In the first frame of images, by manually selecting the target rectangular frame, the target rectangular frame is divided into multiple small local image blocks, and their grayscale histograms are extracted using integral image technology ; Treat the local image blocks obtained in the first frame as foreground image blocks, and use the background modeling method based on the local spatial co-occurrence relationship to model the background around the target rectangular frame; in a new frame of image, Firstly, by matching all the foreground image blocks, a target weight image set is generated; then a robust statistical operator is designed to fuse the target weight image set to determine the position of the target in the current frame; after obtaining the position of the target, use the local The background model divides the image block in the target rectangular frame corresponding to the target position into a foreground image block and a background image block; then update the grayscale histogram model of the foreground image block, and update the local background model at the same time; in each frame of image, All proceed in sequence according to the above process until the end of the tracking video.

Beneficial effect: compared with the prior art, the present invention has the following advantages:

(1) The present invention not only has a vision-based target detection function, but also has a wide signal detection range and complete target information acquisition;

(2) The present invention also has the target detection function based on radar, and the perception information of the object comes from itself, is less affected by the external environment, and has higher reliability and accuracy in depth information acquisition;

(3) The present invention does not require the full participation of staff, and can automatically complete the detection and image tracking of moving targets. The radar measures the distance between the unmanned surface vehicle and the dynamic target in real time, so as to realize the positioning of the dynamic target, and use this as a feedback signal to form a closed-loop control and guide the tracking of the unmanned surface vehicle;

(4) The present invention uses the two sensors of pan-tilt camera and millimeter-wave radar to jointly locate the position of the water surface target, so that the detection and tracking results are more accurate, the detection efficiency is improved, manpower and financial resources are greatly saved, and the market prospect is broad;

(5) The present invention takes the surface target as the research object, detects, recognizes and tracks the target, and has great theoretical and practical significance for shipwreck search and rescue, ocean ship detection and monitoring management, and future new military combat strategies.

Description of drawings

Fig. 1 is a structural block diagram of the present invention;

Fig. 2 is a schematic diagram of the process of processing 2D images in the present invention;

Fig. 3 is a block diagram of texture mode and motion mode extraction in the present invention;

Fig. 4 is a schematic diagram of moving target segmentation in the present invention;

Fig. 5 is a block diagram of the moving target tracking algorithm in the present invention.

Detailed ways

The technical solutions of the present invention will be described in detail below, but the protection scope of the present invention is not limited to the embodiments.

As shown in Fig. 1, a kind of visual detection based on USV of the present invention and surface target tracking system include command center, surface unmanned boat and detection system; command center is the control terminal of water surface unmanned boat, and communicate with each other and send instructions to the surface unmanned boat; the surface unmanned boat receives the information sent from the detection system, and completes the task of tracking the surface target through the navigation system; the detection system detects and locates the target object on the water surface to know its position Coordinates, and send the information to the surface unmanned boat, while waiting for the next command from the command center; the detection system is integrated with a pan-tilt camera and a millimeter-wave radar.

Among them, the PTZ camera obtains the 2D position information of the object in the image, and at the same time, the millimeter-wave radar obtains the distance information of the object. The two cooperate to detect and locate the 3D information of the water surface target, and complete the information output of the water surface target through the navigation system, that is, through the wireless The communication method is transmitted to the surface unmanned vehicle (USV), and the monitoring personnel can monitor the data information received by the USV in real time at the control terminal (ie, the command center), and issue instructions to the USV.

Example 1:

Firstly, the small model of the unmanned boat is placed in the swimming pool, and the command center of the entire surface target tracking system is simulated with a PC. Throw the ball into the swimming pool arbitrarily, and at the same time, the PC sends commands to the unmanned boat to search for spherical floating objects in front. Through spherical target detection, the camera and millimeter-wave radar on the unmanned boat accurately locate the 3D position of the ball and send the information to the navigation system. The final result is that the unmanned boat drives in the direction of the ball, and the search for the target is completed. .

As shown in Figure 2, the detection and tracking method of the USV-based visual detection and water surface target tracking system specifically includes the following steps:

(1) The PTZ camera collects the 2D video image of the surface target, and then processes it;

(11) Preprocess the image sequence captured by the PTZ camera to obtain dynamic scene data suitable for computer processing, and then perform background modeling and detection of dynamic targets, as shown in Figure 3, T in the figure represents T time, because There is a correlation in the spatial domain between adjacent pixels in a dynamic scene, that is, a symbiotic relationship. Therefore, to describe and extract this symbiotic relationship, a moving object detection method based on fusion of texture and motion mode is used: using local binary value mode to extract the texture mode; at the same time, the traditional local binary mode is extended from the spatial domain to the time-space domain to extract the motion mode; each pixel in the scene is modeled separately using the texture mode and the motion model, and then in the classifier The layer will be based on the texture mode and the background model of the motion mode to be fused;

(12) Segmentation of moving targets: as shown in Figure 4, two background clipping images are generated by using the background modeling method based on the Gaussian mixture model and the embedded feature of the adjacent image block, and then these two background clipping images are combined into one The background is cut out from the image; then the connected area of the foreground pixels is extracted, and the pixels in the connected area and its surrounding neighborhood are divided into foreground seed pixels, background seed pixels and unmarked pixels; finally, a closed-form matting algorithm is used to The connected area of the target and its surrounding neighborhood are segmented more finely;

(13) Tracking of the moving target: as shown in Fig. 5, t in the figure represents the image of the tth frame. In the first frame image, by manually selecting the target rectangular frame, the target rectangular frame is divided into multiple small local image blocks, and their grayscale histograms are extracted using the integral image technique; the first frame is divided into The local image blocks are all regarded as foreground image blocks, and the background modeling method based on the local spatial co-occurrence relationship is used to model the background around the target rectangular frame; in a new frame of image, firstly by matching all the foreground image blocks , to generate a target weight image set; then design a robust statistical operator to fuse the target weight image set to determine the position of the target in the current frame; after obtaining the position of the target, use the local background model to convert the target position corresponding to The image blocks in the target rectangular frame are divided into foreground image blocks and background image blocks; then update the grayscale histogram model of the foreground image block, and update the local background model at the same time; when the tracking video is not over, that is, t has not yet reached When tracking the total frame of the video, for each frame of image, the above process is cycled in sequence until the end of the tracking video, and then the tracking of the moving target is completed;

(2) PTZ camera tracking control, real-time adjustment of the pitch and deflection angle of the PTZ camera to keep the target in the center of the image;

(3) Utilize the millimeter-wave radar to further measure the distance between the unmanned surface vehicle and the moving target, and combine the data obtained in step (2) to obtain the 3D data of the moving target to complete the tracking and precise positioning of the moving target;

(4) The navigation system of the unmanned surface craft can autonomously track the moving target on the water surface.

Claims (3)

1., based on vision-based detection and the waterborne target tracing system of USV, it is characterized in that: comprise command centre, unmanned surface vehicle and investigation system;

Described command centre is the control terminal of unmanned surface vehicle, and communicates to connect between unmanned surface vehicle and send instruction to unmanned surface vehicle;

Described unmanned surface vehicle receives the information sent from investigation system, and completes tracking waterborne target task by navigational system;

Described investigation system detects the target object on the water surface and locates knows its position coordinates, and information is sent to unmanned surface vehicle, waits for next step instruction of command centre simultaneously; Investigation system is integrated with monopod video camera and millimetre-wave radar.

2., based on a detection method for tracing for the vision-based detection based on USV according to claim 1 and waterborne target tracing system, it is characterized in that: comprise the following steps:

(1) the 2D video image that monopod video camera photographs is processed;

(2) monopod video camera tracing control, the pitching deflection angle of adjustment monopod video camera, makes target remain on the central authorities of image in real time;

(3) utilize the distance between millimetre-wave radar measurement unmanned water surface ship and moving target, integrating step (2) the data obtained, the 3D data obtaining moving target complete the tracking of moving target and accurately locate;

(4) unmanned water surface ship navigational system carries out autonomous trackable surface moving target traveling.

3. the detection method for tracing of the vision-based detection based on USV according to claim 2 and waterborne target tracing system, it is characterized in that: described step (1) comprises image sequence pre-service, carry out the detection of background modeling and moving target, the segmentation of moving target and motion target tracking four steps, detailed process is as follows:

(11) image sequence pre-service comprises image conversion, image denoising and image enhaucament;

(12) detection of background modeling and dynamic object, the i.e. moving object detection of combined with texture and motion model: use local binary patterns to extract texture pattern; Traditional local binary patterns is extended to time-space domain from spatial domain, for extracting motor pattern, concrete grammar is simultaneously: for extracting the descriptor of texture pattern for (x in t image _t,c, y _t,c) the pixel g at place _t,cconsider its eight neighborhood territory pixel g _t,p, p=0 ... .7, carries out binaryzation by each neighborhood territory pixel and this pixel and compares, obtain the binary string of eight, be i.e. a code word LBP at this pixel place ^t(x _t,c, y _t,c):

${\mathrm{LBP}}^t(x_{t,c},y_{t,c})={\mathrm{\&Sigma;}}_{p=0}^{7}s(g_{t,p}-g_{t,c})2^p$

Wherein $s\left(x\right)=\left\{\begin{array}{cc}1,& x\&GreaterEqual;0\\ 0,& x<0\end{array}\right.$

This code word portrays pixel (x _t,c, y _t,c) a kind of texture pattern of being formed with its surrounding pixel, then respectively modeling is carried out to each pixel employing texture pattern in scene and motion model, then in sorter aspect, the background model based on texture pattern and motor pattern is merged;

(13) segmentation of moving target: by using based on gauss hybrid models and embedding the background modeling method of feature based on neighbour's image block and generate two width backgrounds and wipe out image, then this two width background is wiped out Images uniting one width background and wipe out image; Then extract the connected region of foreground pixel, by connected region and and surrounding neighbors in pixel be divided into foreground seeds pixel, background sub pixel and unmarked pixel; The stingy nomography based on closing form is finally adopted to carry out more meticulous segmentation to the connected region and surrounding neighbors thereof that comprise moving target;

(14) tracking of moving target: in the first two field picture, by artificial selected target rectangle frame, is divided into multiple little topography's block by target rectangle frame, and uses the technology of integrogram to extract their grey level histogram; All be used as divide the topography's block obtained in the first frame as foreground image block, use the background modeling method based on local space symbiosis to carry out modeling to the background around target rectangle frame simultaneously; In a new two field picture, first by the foreground image block that coupling is all, generate a target weight image collection; Then design the Statistical Operator of a robust, merge target weight image collection, thus determine target position in the current frame; Use local background's model after obtaining the position of target, the image block in target rectangle frame corresponding for target location is divided into foreground image block and background image block; Then upgrade the intensity histogram graph model of foreground image block, upgrade local background's model simultaneously; In each two field picture, all carry out successively by said process, terminate until follow the tracks of video.