CN103024350B - A kind of principal and subordinate's tracking of binocular PTZ vision system and the system of application the method - Google Patents

Abstract

The present invention proposes a master-slave tracking method of a binocular PTZ vision system and a system applying the method. The master-slave tracking method includes the following steps: respectively calibrate two PTZ cameras to obtain a camera coordinate system model; establish spherical coordinates system, so that the longitude values of the corresponding points on the spherical coordinate system on the two PTZ camera coordinate systems are consistent, and the latitude difference is used to express the difference in viewing angle, and the transformation matrix is obtained for the two PTZ camera coordinate systems respectively; select the main camera and From the camera, select the tracking target on the observation image of the main camera, so that the main camera can estimate the pan-tilt-zoom parameters of the slave camera under any pan-tilt-zoom parameters (P _M , T _M , Z _M ) Output high-resolution panoramas. The master-slave tracking method of the present invention can not only realize monitoring in a wide range of scenarios, but also reduce hardware overhead. The design of the two PTZ cameras of the master-slave tracking system of the present invention is symmetrical, and the positions of the two cameras can be exchanged according to different tasks, which has strong flexibility and is beneficial to later information fusion.

Description

A master-slave tracking method of a binocular PTZ vision system and a system applying the method

technical field

The invention relates to the technical field of computer vision intelligent monitoring, in particular to a master-slave tracking system and a tracking method using a binocular PTZ camera vision system.

Background technique

With the urgent need for public security and military security worldwide, intelligent visual surveillance technology has received more and more attention. In the early application of intelligent visual surveillance technology, the degree of intelligence was very low, and the security protection function mainly relied on people to judge the events in the surveillance video, and the reliability and automation were very low. With the rapid development of computer processing performance and the continuous improvement of computer vision theory, the intelligent visual monitoring system has been greatly developed, and its development prospects are very broad, which has attracted the attention of many scholars and engineers.

A core task of an intelligent visual surveillance system is to track objects of interest in real time. Traditional intelligent visual monitoring systems mostly use static cameras. Due to the fixed field of view and single resolution of the camera, it is impossible to obtain high-resolution images of tracking targets, which will bring difficulties to future inquiries and proof work. With the improvement of hardware level, Active tracking systems based on monocular PTZ (Pan/Tilt/Zoom, omni-directional movement of the pan and tilt and lens zoom, zoom control) cameras have been widely studied and applied. These systems can make the target appear in the center of the image on a larger scale, However, due to the loss of panoramic information due to the narrow field of view, it is difficult to intuitively obtain the position of the target in the scene.

In response to the above defects, multi-eye vision systems including PTZ cameras have become a research hotspot in intelligent surveillance. These systems generally work in master-slave mode. The master camera realizes the tracking of the target in the panorama, and controls the PTZ camera to actively track the target. The existing master-slave system includes: a system architecture combining a static camera and a PTZ camera. The main problem of this system is that the monitoring field of view is limited, which is limited to the monitoring field of view of the static camera and cannot be adapted to the monitoring of large scenes; The system configuration combining multiple still cameras and PTZ cameras, although this system configuration is to expand the monitoring range, greatly increases the hardware overhead; the tracking system configuration combining omnidirectional cameras and PTZ cameras, this system is generally used in indoor scenes, Moreover, due to the low resolution of the omnidirectional camera, it is difficult to realize information fusion between the two cameras. Therefore, it is an urgent technical problem to find a visual tracking method that can realize monitoring in a wide range of scenes and reduce hardware overhead.

Contents of the invention

The present invention aims at at least solving the technical problems existing in the prior art, and particularly innovatively proposes a master-slave tracking system and a tracking method of a binocular PTZ vision system.

In order to achieve the above object of the present invention, according to one aspect of the present invention, the present invention provides a master-slave tracking method of a binocular PTZ vision system, comprising the steps of:

S1: Calibrate the two PTZ cameras respectively to obtain the camera coordinate system model;

S2: Establish a spherical coordinate system, so that the longitude values of the corresponding points on the two PTZ camera coordinate systems on the spherical coordinate system are consistent, and the latitude difference is used to express the difference in viewing angle, and the transformation is obtained for the two PTZ camera coordinate systems respectively matrix;

S3: Select the main camera and the secondary camera, and select the tracking target on the observation image of the main camera, so that the main camera is under any pan-tilt-zoom parameters (P _M , T _M , Z _M ), and the tracking target is at Observe the motion track on the image I _M from the camera Estimated pan-tilt-zoom parameters from the camera $(P_S^t,T_S^t,Z_S^t)(t=1...\mathrm{no});$

S4: Output a high-resolution panorama.

The master-slave tracking method of the binocular PTZ vision system of the present invention can not only realize monitoring in a large-scale scene, but also reduce hardware overhead.

In a preferred embodiment of the present invention, any one of the two cameras is used as the master camera, and the other one is used as the slave camera.

In another preferred embodiment of the present invention, the two cameras perform visual inspections in the monitoring scene respectively according to the monitoring tasks, and when any one of the cameras finds suspicious objects, it acts as the main camera and adopts the tracking method under a static camera to carry out visual inspection on it. Tracking, and enter the master-slave working mode, control another camera as a slave camera to actively track suspicious objects.

In the monitoring of a large-scale scene, the invention realizes the master-slave mode tracking under any parameter of two cameras, and obtains a high-resolution panoramic image of the tracking target. Due to the different pan-tilt-zoom parameters corresponding to different monitoring scenarios or monitoring tasks in practical applications, realizing master-slave mode tracking under any parameter has greater practical application value and is more versatile. The present invention only needs two The camera is installed fixedly, which can conveniently handle the situation where the parameters of the two cameras change, and is not affected by the change of the camera parameters. At the same time, the present invention uses high-resolution panorama as the result output, while displaying the high-resolution target image, it can more intuitively display the motion information of the target in the entire monitoring scene, which enhances visibility and practicability, and can be used in later stages Target behavior analysis, gesture recognition, gait analysis, etc.

In order to achieve the above object of the present invention, according to two aspects of the present invention, the present invention provides a master-slave tracking system of a binocular PTZ vision system, including a host; a first camera and a second camera, the first camera and The second cameras are respectively connected to the host computer, any one of the two cameras is used as the main camera, and the other is used as the slave camera, and an image capture module, and the image capture module performs information interaction with the first camera and the second camera respectively; A first camera control module and a second camera control module, the first camera control module controls the operation and observation mode of the first camera, and the second camera control module controls the operation and observation mode of the second camera; the first camera control module controls the operation and observation mode of the second camera; The camera and the second camera also respectively include a target tracking module, a target prediction module, a spherical coordinate model module and a PTZ parameter module. In the main camera, the target tracking module tracks suspicious objects and transmits the tracking results to The target prediction module, the target prediction module is connected with the spherical coordinate model module of the slave camera, the target prediction module predicts the position of the suspicious object, and transmits the prediction result to the slave camera, the spherical coordinate of the slave camera The model module transforms the coordinates, and transmits the transformation result to the PTZ parameter module to calculate the PTZ parameters, and the PTZ parameter module transmits the calculated PTZ parameters to the corresponding camera control module.

The design of the two PTZ cameras of the master-slave tracking system of the binocular PTZ vision system of the present invention is symmetrical, and the positions of the two can be interchanged according to different tasks, which has strong flexibility and is beneficial to later information fusion.

Additional aspects and advantages of the invention will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention.

Description of drawings

The above and/or additional aspects and advantages of the present invention will become apparent and comprehensible from the description of the embodiments in conjunction with the following drawings, wherein:

Fig. 1 is the structural representation of the master-slave tracking system of the binocular PTZ vision system of the present invention;

Fig. 2 is a schematic diagram of the PTZ parameter geometric relationship of two cameras of the present invention;

Fig. 3 is the flow chart of the present invention estimation from camera parameter;

Fig. 4 is a schematic diagram of estimating the latitude range from the camera according to the depth range of the scene;

Figure 5 is a schematic diagram of Pan-tilt parameter estimation;

Fig. 6 is master-slave tracking and high-resolution panorama result figure in the first preferred embodiment of the present invention;

Fig. 7 is a result diagram of master-slave tracking and high-resolution panorama in the second preferred embodiment of the present invention.

Detailed ways

Embodiments of the present invention are described in detail below, examples of which are shown in the drawings, wherein the same or similar reference numerals designate the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by referring to the figures are exemplary only for explaining the present invention and should not be construed as limiting the present invention.

In the description of the present invention, unless otherwise specified and limited, it should be noted that the terms "installation", "connection" and "connection" should be understood in a broad sense, for example, it can be mechanical connection or electrical connection, or two The internal communication of each element may be directly connected or indirectly connected through an intermediary. Those skilled in the art can understand the specific meanings of the above terms according to specific situations.

The present invention provides a master-slave tracking system of a binocular PTZ vision system, as shown in Figure 1, including a host (not shown in the figure), a first camera and a second camera, the first camera and the second camera are respectively It is connected with the host, and any one of the two cameras is used as the main camera, and the other is used as the slave camera; the system also includes an image capture module, a first camera control module and a second camera control module, and the image capture module is connected to the first camera respectively. The camera and the second camera carry out information interaction; the first camera control module controls the operation and observation mode of the first camera, and the second camera control module controls the operation and observation mode of the second camera; the first camera and the second camera also respectively It includes a target tracking module, a target prediction module, a spherical coordinate model module and a PTZ parameter module. In the main camera, the target tracking module tracks suspicious objects and transmits the tracking result to the target prediction module. The target prediction module communicates with the slave camera The spherical coordinate model module of the camera is connected, the target prediction module predicts the position of the suspicious object, and transmits the prediction result to the slave camera, the spherical coordinate model module of the slave camera transforms the coordinates, and transmits the transformation result to the PTZ parameter module to calculate the PTZ parameters, the PTZ parameter module transmits the calculated PTZ parameters to the corresponding camera control module.

According to the monitoring tasks, the two cameras of the present invention carry out visual inspection in the monitoring scene respectively. When any camera finds suspicious objects, it will be used as the main camera to track it by using the tracking method under the static camera, and enter the master-slave working mode to control Another camera acts as a slave camera to actively track suspicious objects.

The present invention adopts binocular PTZ cameras to realize master-slave mode tracking. The reason for choosing such a hardware combination is that PTZ cameras are the simplest active cameras, and the angle of view can be changed by changing the pan and tilt parameters, or by changing the zoom Parameters to change the image resolution, its integration is very high, there are more mature products, with the reduction of hardware costs has been more and more used in practical applications. In addition, two PTZ cameras are used. Due to the symmetry of the two cameras, the positions of the two cameras can be interchanged according to different tasks, which is flexible and beneficial to later information fusion. At the same time, due to the variability and controllability of camera parameters, the monitoring field of view can be switched by switching the pan-tilt-zoom parameter to achieve the purpose of obtaining a larger monitoring field of view with the fewest cameras.

The present invention also provides a master-slave tracking method of a binocular PTZ vision system, which can, but is not limited to, adopt the tracking system shown in Figure 1 of the present invention, and the master-slave tracking method comprises the following steps:

S1: Calibrate the two PTZ cameras respectively to obtain the camera coordinate system model;

S2: Establish a spherical coordinate system, so that the longitude values of the corresponding points on the spherical coordinate system on the two PTZ camera coordinate systems are consistent, and the latitude difference is used to express the difference in viewing angle, and the transformation matrix is respectively obtained for the two PTZ camera coordinate systems;

S3: Select the main camera and the secondary camera, and select the tracking target on the observation image of the main camera, so that the main camera is under any pan-tilt-zoom parameters (P _M , T _M , Z _M ), and the tracking target is at Observe the motion track on the image I _M from the camera Estimated pan-tilt-zoom parameters from the camera $(P_S^t,T_S^t,Z_S^t)(t=1...\mathrm{no});$

S4: Output a high-resolution panorama.

In a preferred embodiment of the present invention, the present invention carries out the following steps sequentially in the host computer:

In the first step, the two PTZ cameras are calibrated separately to obtain the camera coordinate system model. In this embodiment, the method of feature point matching is used to calibrate the external parameter matrix R and the internal parameter matrix K respectively for the two PTZ cameras, wherein the external parameter matrix R is only related to the pan and tilt parameters, and the rotation axes of the pan and tilt parameters perpendicular to the camera center. The steps to calibrate the internal reference matrix K are:

First, set the pixel aspect ratio _ar and inclination S of the camera. In this embodiment, the pixel aspect ratio _ar =1 and the inclination S=0 of the camera are set, and the influence of image distortion is not considered;

Then, use the zoom center instead of the main point (u ₀ , v ₀ ), keep the pan and tilt parameters of the camera unchanged, obtain the image sequence by changing the zoom parameter, extract SIFT feature points from each image, and compare two adjacent frames The feature points in the image are matched, and finally the zoom center is obtained by the least square method;

Finally, construct a model of the focal length f varying with the zoom parameter. For each fixed zoom parameter value, multiple images are obtained by changing the pan and tilt parameters a small amount, and the focal length f is estimated through image registration, using multiple sets of discrete sampling values {zoom, f}Construct model f=f _z (zoom).

Through the above steps, the coordinate system model of the PTZ camera can be expressed as:

$\stackrel{<span\; class="notranslate">\; ~</span>}{\mathrm{<span\; class="notranslate">\; x</span>}}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; \&kappa;K</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; zoom</span>}<span\; class="notranslate">\; )</span>\mathrm{<span\; class="notranslate">\; R</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; the\; pan</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; tilt</span>}<span\; class="notranslate">\; )</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; \&kappa;</span>}\left[\begin{array}{ccc}{\mathrm{<span\; class="notranslate">\; f</span>}}_{\mathrm{<span\; class="notranslate">\; z</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; zoom</span>}<span\; class="notranslate">\; )</span>& \mathrm{<span\; class="notranslate">\; 0</span>}& {\mathrm{<span\; class="notranslate">\; u</span>}}_{\mathrm{<span\; class="notranslate">\; 0</span>}}\\ \mathrm{<span\; class="notranslate">\; 0</span>}& {\mathrm{<span\; class="notranslate">\; f</span>}}_{\mathrm{<span\; class="notranslate">\; z</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; zoom</span>}<span\; class="notranslate">\; )</span>& {\mathrm{<span\; class="notranslate">\; v</span>}}_{\mathrm{<span\; class="notranslate">\; 0</span>}}\\ \mathrm{<span\; class="notranslate">\; 0</span>}& \mathrm{<span\; class="notranslate">\; 0</span>}& \mathrm{<span\; class="notranslate">\; 1</span>}\end{array}\right]\left[\begin{array}{ccc}\mathrm{<span\; class="notranslate">\; cos</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; the\; pan</span>}<span\; class="notranslate">\; )</span>& \mathrm{<span\; class="notranslate">\; 0</span>}& \mathrm{<span\; class="notranslate">\; sin</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; the\; pan</span>}<span\; class="notranslate">\; )</span>\\ <span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; sin</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; the\; pan</span>}<span\; class="notranslate">\; )</span>\mathrm{<span\; class="notranslate">\; sin</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; tilt</span>}<span\; class="notranslate">\; )</span>& \mathrm{<span\; class="notranslate">\; cos</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; tilt</span>}<span\; class="notranslate">\; )</span>& \mathrm{<span\; class="notranslate">\; cos</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; the\; pan</span>}<span\; class="notranslate">\; )</span>\mathrm{<span\; class="notranslate">\; sin</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; tilt</span>}<span\; class="notranslate">\; )</span>\\ <span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; sin</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; the\; pan</span>}<span\; class="notranslate">\; )</span>\mathrm{<span\; class="notranslate">\; cos</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; tilt</span>}<span\; class="notranslate">\; )</span>& <span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; sin</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; tilt</span>}<span\; class="notranslate">\; )</span>& \mathrm{<span\; class="notranslate">\; cos</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; the\; pan</span>}<span\; class="notranslate">\; )</span>\mathrm{<span\; class="notranslate">\; cos</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; tilt</span>}<span\; class="notranslate">\; )</span>\end{array}\right]\mathrm{<span\; class="notranslate">\; x</span>}$

Among them, κ is the scale factor, Represents the homogeneous coordinates of the image, and X represents the coordinates of the corresponding camera system.

The second step: establish a spherical coordinate system, so that the longitude values of the corresponding points on the two PTZ camera coordinate systems on the spherical coordinate system are consistent, and the latitude difference is used to express the difference in viewing angle, and the transformation is obtained separately for the two PTZ camera coordinate systems Matrix, the specific steps are as follows:

First, manually collect N pairs of images of two cameras in the same monitoring scene, so that the field of view of each pair of images is basically consistent, so as to obtain more matching feature point pairs, and make the collected images cover the entire monitoring scene as much as possible .

Then, perform SIFT feature point extraction and matching on each pair of images, and then use the RANSAC method to estimate the fundamental matrix F _j (j=1,...,N) of each pair of images, and record the matched feature point pairs Among them, j is the jth pair of images of N pairs of images, and k is the kth pair of feature points of the jth pair of images.

Then, estimate the pole pair {E ₁ ,E ₂ } of the spherical coordinate system, let as well as then there is $f_j^{\left(1\right)}{\mathrm{E.}}_1={\mathrm{E.}}_2^T{f}_j^{\left(2\right)}=0,$ definition $\left\{\begin{array}{c}{A}_1={[{\left(f_1^{\left(1\right)}\right)}^T,{\left(f_2^{\left(1\right)}\right)}^T,...,{\left(f_N^{\left(1\right)}\right)}^T]}^T\\ A_2=[f_1^{\left(2\right)},f_2^{\left(2\right)},...,f_N^{\left(2\right)}]\end{array}\right.,$ Then use the singular value decomposition method to solve A ₁ E ₁ =0 and Get {E ₁ ,E ₂ }.

Then, estimate the reference zero longitude line of the spherical coordinate system, use the camera imaging model obtained in the first step, and use the matching feature point pairs of each pair of images obtained Transformed into the respective camera coordinate system, we get On the spherical coordinate system of the two cameras, an isolongitude line is arbitrarily selected as the reference zero longitude line, and in the spherical coordinate system determined by it and the obtained pole, the Convert to longitude and latitude coordinates, calculate the absolute deviation of the longitude components of all feature points, adjust one of the reference zero longitude lines to minimize the average deviation, and then obtain the final reference zero longitude lines n _{1_ref} and n _{2_ref} .

Finally, set Cross_E ₁ _n _{1_ref} =E ₁ ×n _{1_ref} and Cross_E ₂ _n _{2_ref} =E ₂ ×n _{2_ref} to obtain transformation matrices R ₁ , R ₂ , R ₁ =[E ₁ n _{1_ref} Cross_E ₁ _n _{1_ref} ], R ₂ =[E ₂ n _{2_ref} Cross_E ₂ _n _{2_ref} ].

Step 3: Select the main camera and the slave camera, and select the tracking target on the observation image of the main camera, so that the main camera can be tracked by any pan-tilt-zoom parameters (P _M , T _M , Z _M The trajectory of the target on the image I _M observed from the camera Estimated pan-tilt-zoom parameters from the camera $(P_S^t,T_S^t,Z_S^t)(t=1...\mathrm{no}).$

Since the first step and the second step can generally be performed offline, as long as the two cameras are fixed, the estimated transformation matrices R ₁ and R ₂ will not change. In actual online application, due to the symmetry of the two cameras, both can be used as the main camera. When any one of the two cameras is used as the main camera, the other one is used as the slave camera. Carry out visual inspection in the scene, when any camera finds suspicious objects, it will act as the main camera to track it using the tracking method under the static camera, and enter the master-slave working mode, control another camera as a slave camera to actively track suspicious objects . In the present invention, for the convenience of expression, any camera is used as the main camera, which is recorded as Cam-M, and the other camera is the slave camera, which is recorded as Cam-S, and the transformation from the camera coordinate system corresponding to Cam-M to the spherical coordinate system The matrix is R _M , the transformation matrix from the camera coordinate system corresponding to Cam-S to the spherical coordinate system is R _S , the schematic diagram of the geometric relationship between the two cameras is shown in Figure 2, and I _M is assumed to be the main camera Cam-M in parameters (P _M , T _M , Z _M ) observation image, To track the observation of the target P on the image I _M at time t, For Cam-S to make the tracking target P in its observation image at time t The parameters of the center, the specific operation steps of the third step are as follows:

First, select the tracking target on the observation image I _M of the main camera Cam-M, and use the Mean-shift tracking algorithm provided by Opencv to track the target frame by frame. The filtering method predicts the position of the target, and sets the predicted value of tracking the center position of the target at time t as

Then, estimate the parameter value from the camera Cam-S at each moment As shown in Figure 3, the steps are as follows:

Step 1, transform the observed image coordinates of the main camera Cam-M into camera coordinates, set is the image coordinate corresponding to the predicted value of the tracking target at time t, using the monocular PTZ camera coordinate system model obtained in the first step, it can be calculated A ray under the determined camera coordinate system is as follows:

${\mathrm{<span\; class="notranslate">\; Y</span>}}_{\mathrm{<span\; class="notranslate">\; m</span>}}^{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; \&kappa;</span>}{\mathrm{<span\; class="notranslate">\; R</span>}}^{<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; P</span>}}_{\mathrm{<span\; class="notranslate">\; m</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; T</span>}}_{\mathrm{<span\; class="notranslate">\; m</span>}}<span\; class="notranslate">\; )</span>{\mathrm{<span\; class="notranslate">\; K</span>}}^{<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; Z</span>}}_{\mathrm{<span\; class="notranslate">\; m</span>}}<span\; class="notranslate">\; )</span>\stackrel{<span\; class="notranslate">\; ~</span>}{{\mathrm{<span\; class="notranslate">\; p</span>}}_{\mathrm{<span\; class="notranslate">\; m</span>}<span\; class="notranslate">\; \&Right\; Arrow;</span>\mathrm{<span\; class="notranslate">\; P</span>}}^{\mathrm{<span\; class="notranslate">\; t</span>}}}$

in, is the normalized coordinate of the predicted value of the tracking target at time t in the camera coordinate system of the main camera, Represents homogeneous coordinates, κ is a scale factor, to satisfy (P _M , T _M , Z _M ) represent the parameters of the main camera Cam-M.

Step 2, transform the camera coordinates of the main camera to the spherical coordinates of the main camera, and set Transform to the spherical coordinate system corresponding to the main camera Cam-M, and calculate the longitude, latitude and longitude of the predicted value of the tracking target at time t in the spherical coordinate system of the main camera and latitude Calculated as follows:

${\mathrm{<span\; class="notranslate">\; Y</span>}}_{\mathrm{<span\; class="notranslate">\; m</span>}<span\; class="notranslate">\; \&Right\; Arrow;</span>\mathrm{<span\; class="notranslate">\; r</span>}}^{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; =</span>{\mathrm{<span\; class="notranslate">\; R</span>}}_{\mathrm{<span\; class="notranslate">\; m</span>}}{\mathrm{<span\; class="notranslate">\; Y</span>}}_{\mathrm{<span\; class="notranslate">\; m</span>}}^{\mathrm{<span\; class="notranslate">\; t</span>}}$

${\mathrm{<span\; class="notranslate">\; \&alpha;</span>}}_{\mathrm{<span\; class="notranslate">\; m</span>}}^{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; a</span>}\mathrm{<span\; class="notranslate">\; the\; tan</span>}\mathrm{<span\; class="notranslate">\; 2</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; Y</span>}}_{\mathrm{<span\; class="notranslate">\; m</span>}<span\; class="notranslate">\; \&Right\; Arrow;</span>\mathrm{<span\; class="notranslate">\; r</span>}}^{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 3</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; Y</span>}}_{\mathrm{<span\; class="notranslate">\; m</span>}<span\; class="notranslate">\; \&Right\; Arrow;</span>\mathrm{<span\; class="notranslate">\; r</span>}}^{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 2</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; )</span>$

${\mathrm{<span\; class="notranslate">\; \&beta;</span>}}_{\mathrm{<span\; class="notranslate">\; m</span>}}^{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; a</span>}\mathrm{<span\; class="notranslate">\; cos</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; Y</span>}}_{\mathrm{<span\; class="notranslate">\; m</span>}<span\; class="notranslate">\; \&Right\; Arrow;</span>\mathrm{<span\; class="notranslate">\; r</span>}}^{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; )</span>$

Among them, R _M is the transformation matrix from the camera coordinate system of the main camera Cam-M to the spherical coordinate system, for Cartesian coordinates on the corresponding spherical coordinate system, representation vector The nth element of .

Step 3. Map the spherical coordinates of the main camera Cam-M to the spherical coordinate system of the slave camera Cam-S. The purpose of this step is to obtain the latitude and longitude in the spherical coordinate system of the main camera Cam-M Estimate the latitude and longitude of the corresponding point in the spherical coordinate system from the camera Cam-S Since the spherical coordinate system established in the second step has made the longitude values of the corresponding points in the two camera coordinate systems consistent in their respective spherical coordinate systems, so to estimate Assuming that the target depth at time t is known, denoted by D _t , the depth is defined as the distance from the target P to the baseline O _M O _S at time t, which intersects with the baseline at point O ^t , and b is the baseline length of the two cameras, as shown in Figure 3 shown.

set up Then there is the following geometric relationship:

${\mathrm{<span\; class="notranslate">\; the\; tan</span>}\mathrm{<span\; class="notranslate">\; \&beta;</span>}}_{\mathrm{<span\; class="notranslate">\; m</span>}}^{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; =</span>\frac{{\mathrm{<span\; class="notranslate">\; D.</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}}}{{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; m</span>}}^{\mathrm{<span\; class="notranslate">\; t</span>}}}$

$\mathrm{<span\; class="notranslate">\; the\; tan</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; \&pi;</span>}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; \&beta;</span>}}_{\mathrm{<span\; class="notranslate">\; S</span>}}^{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\frac{{\mathrm{<span\; class="notranslate">\; D.</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}}}{{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; S</span>}}^{\mathrm{<span\; class="notranslate">\; t</span>}}}$

$\mathrm{<span\; class="notranslate">\; b</span>}<span\; class="notranslate">\; =</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; m</span>}}^{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; S</span>}}^{\mathrm{<span\; class="notranslate">\; t</span>}}$

available:

${\mathrm{<span\; class="notranslate">\; \&beta;</span>}}_{\mathrm{<span\; class="notranslate">\; S</span>}}^{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; a</span>}\mathrm{<span\; class="notranslate">\; the\; tan</span>}<span\; class="notranslate">\; (</span>\frac{{<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; D.</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; \&CenterDot;</span>\mathrm{<span\; class="notranslate">\; the\; tan</span>}{\mathrm{<span\; class="notranslate">\; \&beta;</span>}}_{\mathrm{<span\; class="notranslate">\; m</span>}}^{\mathrm{<span\; class="notranslate">\; t</span>}}}{\mathrm{<span\; class="notranslate">\; b</span>}<span\; class="notranslate">\; \&CenterDot;</span>{\mathrm{<span\; class="notranslate">\; the\; tan</span>}\mathrm{<span\; class="notranslate">\; \&beta;</span>}}_{\mathrm{<span\; class="notranslate">\; m</span>}}^{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; D.</span>}}_{\mathrm{<span\; class="notranslate">\; t</span>}}}<span\; class="notranslate">\; )</span>$

Since the depth D _t of the target in the scene is unknown, it is impossible to accurately estimate the latitude of the target in the spherical coordinate system of the camera Cam-S In this embodiment, due to the calculation of the target depth D _t and the latitude of the calculation target in the spherical coordinate system from the camera Cam-S It is basically equivalent and can be solved as a chicken-and-egg problem. Considering that in large scene monitoring, the baseline length b of the two cameras is very small relative to the scene depth, so the scene depth ranges D _min and D _max can be given to estimate the corresponding latitudes respectively and As shown in Figure 4.

Since the depth of the scene is much greater than the distance between the two cameras, and The difference between them is very small. In addition, considering the real-time requirements in the master-slave tracking, the linear weighted method is used to estimate the latitude value of the slave camera The calculation formula is as follows:

${\mathrm{<span\; class="notranslate">\; \&beta;</span>}}_{\mathrm{<span\; class="notranslate">\; S</span>}<span\; class="notranslate">\; \&Right\; Arrow;</span>\mathrm{<span\; class="notranslate">\; max</span>}}^{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; a</span>}\mathrm{<span\; class="notranslate">\; the\; tan</span>}<span\; class="notranslate">\; (</span>\frac{{<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; D.</span>}}_{\mathrm{<span\; class="notranslate">\; min</span>}}<span\; class="notranslate">\; \&Center\; Dot;</span>{\mathrm{<span\; class="notranslate">\; the\; tan</span>}\mathrm{<span\; class="notranslate">\; \&beta;</span>}}_{\mathrm{<span\; class="notranslate">\; m</span>}}^{\mathrm{<span\; class="notranslate">\; t</span>}}}{\mathrm{<span\; class="notranslate">\; b</span>}<span\; class="notranslate">\; \&Center\; Dot;</span>{\mathrm{<span\; class="notranslate">\; the\; tan</span>}\mathrm{<span\; class="notranslate">\; \&beta;</span>}}_{\mathrm{<span\; class="notranslate">\; m</span>}}^{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; D.</span>}}_{\mathrm{<span\; class="notranslate">\; min</span>}}}<span\; class="notranslate">\; )</span>$

${\mathrm{<span\; class="notranslate">\; \&beta;</span>}}_{\mathrm{<span\; class="notranslate">\; S</span>}<span\; class="notranslate">\; \&Right\; Arrow;</span>\mathrm{<span\; class="notranslate">\; min</span>}}^{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; a</span>}\mathrm{<span\; class="notranslate">\; the\; tan</span>}<span\; class="notranslate">\; (</span>\frac{{<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; D.</span>}}_{\mathrm{<span\; class="notranslate">\; max</span>}}<span\; class="notranslate">\; \&Center\; Dot;</span>{\mathrm{<span\; class="notranslate">\; the\; tan</span>}\mathrm{<span\; class="notranslate">\; \&beta;</span>}}_{\mathrm{<span\; class="notranslate">\; m</span>}}^{\mathrm{<span\; class="notranslate">\; t</span>}}}{\mathrm{<span\; class="notranslate">\; b</span>}<span\; class="notranslate">\; \&CenterDot;</span>{\mathrm{<span\; class="notranslate">\; the\; tan</span>}\mathrm{<span\; class="notranslate">\; \&beta;</span>}}_{\mathrm{<span\; class="notranslate">\; m</span>}}^{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; D.</span>}}_{\mathrm{<span\; class="notranslate">\; max</span>}}}<span\; class="notranslate">\; )</span>$

${\mathrm{<span\; class="notranslate">\; \&beta;</span>}}_{\mathrm{<span\; class="notranslate">\; S</span>}}^{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; \&lambda;</span>}<span\; class="notranslate">\; \&Center\; Dot;</span>{\mathrm{<span\; class="notranslate">\; \&beta;</span>}}_{\mathrm{<span\; class="notranslate">\; S</span>}<span\; class="notranslate">\; \&Right\; Arrow;</span>\mathrm{<span\; class="notranslate">\; max</span>}}^{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; \&gamma;</span>}<span\; class="notranslate">\; \&Center\; Dot;</span>{\mathrm{<span\; class="notranslate">\; \&beta;</span>}}_{\mathrm{<span\; class="notranslate">\; S</span>}<span\; class="notranslate">\; \&Right\; Arrow;</span>\mathrm{<span\; class="notranslate">\; min</span>}}^{\mathrm{<span\; class="notranslate">\; t</span>}}$

Among them, λ and γ are weighting coefficients, satisfying λ+γ=1.

Step 4, transform the surface coordinates of the slave camera Cam-S to the camera coordinates of the slave camera and estimate the parameters of the slave camera First, the latitude and longitude coordinates from the camera Cam-S spherical coordinate system and Transform to the camera coordinate system from the camera Cam-S, this process is similar to the inverse process of step 2, as follows, where is the Cartesian coordinates of the target in the spherical coordinate system of the slave camera Cam-S, is the inverse matrix of the transformation matrix R _S from the camera coordinate system of the camera Cam-S to the spherical coordinate system, is the coordinates of the target in the Cam-S camera coordinate system:

${\mathrm{<span\; class="notranslate">\; Y</span>}}_{\mathrm{<span\; class="notranslate">\; S</span>}<span\; class="notranslate">\; \&Right\; Arrow;</span>\mathrm{<span\; class="notranslate">\; r</span>}}^{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; cos</span>}{\mathrm{<span\; class="notranslate">\; \&beta;</span>}}_{\mathrm{<span\; class="notranslate">\; S</span>}}^{\mathrm{<span\; class="notranslate">\; t</span>}}$

${\mathrm{<span\; class="notranslate">\; Y</span>}}_{\mathrm{<span\; class="notranslate">\; S</span>}<span\; class="notranslate">\; \&Right\; Arrow;</span>\mathrm{<span\; class="notranslate">\; r</span>}}^{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 2</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; sin</span>}{\mathrm{<span\; class="notranslate">\; \&beta;</span>}}_{\mathrm{<span\; class="notranslate">\; S</span>}}^{\mathrm{<span\; class="notranslate">\; t</span>}}\mathrm{<span\; class="notranslate">\; cos</span>}{\mathrm{<span\; class="notranslate">\; \&alpha;</span>}}_{\mathrm{<span\; class="notranslate">\; S</span>}}^{\mathrm{<span\; class="notranslate">\; t</span>}}$

${\mathrm{<span\; class="notranslate">\; Y</span>}}_{\mathrm{<span\; class="notranslate">\; S</span>}<span\; class="notranslate">\; \&Right\; Arrow;</span>\mathrm{<span\; class="notranslate">\; r</span>}}^{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 3</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>{\mathrm{<span\; class="notranslate">\; sin</span>}\mathrm{<span\; class="notranslate">\; \&beta;</span>}}_{\mathrm{<span\; class="notranslate">\; S</span>}}^{\mathrm{<span\; class="notranslate">\; t</span>}}\mathrm{<span\; class="notranslate">\; sin</span>}{\mathrm{<span\; class="notranslate">\; \&alpha;</span>}}_{\mathrm{<span\; class="notranslate">\; S</span>}}^{\mathrm{<span\; class="notranslate">\; t</span>}}$

${\mathrm{<span\; class="notranslate">\; Y</span>}}_{\mathrm{<span\; class="notranslate">\; S</span>}}^{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; =</span>{\mathrm{<span\; class="notranslate">\; R</span>}}_{\mathrm{<span\; class="notranslate">\; S</span>}}^{<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; Y</span>}}_{\mathrm{<span\; class="notranslate">\; S</span>}<span\; class="notranslate">\; \&Right\; Arrow;</span>\mathrm{<span\; class="notranslate">\; r</span>}}^{\mathrm{<span\; class="notranslate">\; t</span>}}$

in, is the Cartesian coordinates of the predicted value of the tracking target at time t in the spherical coordinate system of the camera.

According to the physical meaning of the pan-tilt parameter in the camera coordinate system, the observation point in the camera coordinate system from the camera Cam-S is given Can calculate its corresponding pan-tilt parameter and When the camera Cam-S moves to this parameter value, click The observation on the image is located at the principal point position (that is, the optical axis passes through ), as shown in Figure 5, for the zoom parameter The value can be selected according to the actual application. In this embodiment, the preferred value is

Finally, during the tracking process, the distance between the center position of the tracking target and the image boundary of the main camera is calculated. When the distance is less than the threshold T, the tracking ends and the two cameras return to their preset positions.

Step 4: Output high-resolution panorama, set Capture an image for the main camera Cam-M at time t, An image is captured from camera Cam-S for time instant t. Specifically include the following steps:

First, the image of the main camera Cam-M Find the background model The update formula at pixel (x,y) is:

${\mathrm{<span\; class="notranslate">\; I</span>}}_{\mathrm{<span\; class="notranslate">\; MB</span>}}^{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; \&alpha;</span>}<span\; class="notranslate">\; )</span>{\mathrm{<span\; class="notranslate">\; I</span>}}_{\mathrm{<span\; class="notranslate">\; MB</span>}}^{\mathrm{<span\; class="notranslate">\; t</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; \&alpha;</span>}{\mathrm{<span\; class="notranslate">\; I</span>}}_{\mathrm{<span\; class="notranslate">\; m</span>}}^{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; )</span>$

Among them, α is the update coefficient, which is taken as 0.05 in this embodiment, and the background model at the initial moment is set $I_{\mathrm{MB}}^0(x,\mathrm{the\; y})=I_m^0(x,\mathrm{the\; y}),$ if $|I_m^t(x,\mathrm{the\; y})-I_{\mathrm{MB}}^t(x,\mathrm{the\; y})|>\mathrm{the\; th},$ but If (x, y) belongs to the foreground area, otherwise, the pixel belongs to the background area, where th is the comparison threshold, and in this embodiment, th is set to 20.

Then, the main camera Cam-M at time t is used to capture images based on the combination of feature points and direct pixels and the image captured from the camera at time t For registration, since the baseline length is negligible relative to the depth of the monitoring scene, the registration model can choose an affine model.

Subsequently, the model is registered through the obtained synchronous frame and the obtained main camera Cam-M image The background area and the foreground area, calculate background area and foreground area; then, set the size of the high-resolution panorama I _H to k times the size of the original image, set Represents the main camera low-resolution image and high resolution images from the camera registration model, then I _H and The registration model between is:

${\mathrm{<span\; class="notranslate">\; A</span>}}^{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; =</span>{\mathrm{<span\; class="notranslate">\; A</span>}}_{\mathrm{<span\; class="notranslate">\; MS</span>}}^{\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; \&times;</span>\left[\begin{array}{ccc}\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; /</span>\mathrm{<span\; class="notranslate">\; k</span>}& \mathrm{<span\; class="notranslate">\; 0</span>}& \mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; /</span>\mathrm{<span\; class="notranslate">\; k</span>}\\ \mathrm{<span\; class="notranslate">\; 0</span>}& \mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; /</span>\mathrm{<span\; class="notranslate">\; k</span>}& \mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; /</span>\mathrm{<span\; class="notranslate">\; k</span>}\\ \mathrm{<span\; class="notranslate">\; 0</span>}& \mathrm{<span\; class="notranslate">\; 0</span>}& \mathrm{<span\; class="notranslate">\; 1</span>}\end{array}\right]$

In turn, the high-resolution images from the camera The background area of is mapped to I _H , and the overlapping area is updated with an attenuation factor of 0.5;

Finally, the high-resolution image from the camera is sequentially The foreground region of is mapped to _IH to generate a high-resolution panorama at each moment. Fig. 6 and Fig. 7 are master-slave tracking and high-resolution panorama result figure in the preferred embodiment of the present invention, in Fig. 6 a, left camera is as master camera, right camera is as slave camera, and the parameter of master camera is pan=- 63.52, tilt=-11.32, zoom=11.50, Figure 6b is a high-resolution panorama obtained by mapping the four frames of foreground from the camera to a background. In Figure 7a, the right camera is used as the main camera, and the left camera is used as the slave camera. The parameters of the master camera are pan=-90.66, tilt=-13.50, and zoom=10.00. Figure 7b maps the four frames of the foreground from the camera to a A high-resolution panorama is obtained in the background.

Compared with the master-slave tracking system of static and active cameras, the present invention can increase the monitoring range; compared with the system of multiple static and active cameras, the hardware cost can be reduced; compared with the system of omnidirectional plus active cameras, it is more conducive to information fusion. The invention designs a master-slave control method based on a spherical coordinate model, which can conveniently realize the master-slave mode tracking of two cameras under any pan-tilt-zoom parameter, and realize multi-scale visual attention to the target; The registration model between high-resolution videos adopts a layered processing method to obtain high-resolution panoramic output. The processed results can be used in applications such as criminal evidence collection, surveillance record keeping, and behavior analysis of moving targets.

In the description of this specification, descriptions referring to the terms "one embodiment", "some embodiments", "example", "specific examples", or "some examples" mean that specific features described in connection with the embodiment or example , structure, material or characteristic is included in at least one embodiment or example of the present invention. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

Although the embodiments of the present invention have been shown and described, those skilled in the art can understand that various changes, modifications, substitutions and modifications can be made to these embodiments without departing from the principle and spirit of the present invention. The scope of the invention is defined by the claims and their equivalents.

Claims (7)

1. A master-slave tracking method of a binocular PTZ visual system is characterized by comprising the following steps:

s1: respectively calibrating the two PTZ cameras to obtain a camera coordinate system model;

s2: establishing a spherical coordinate system, keeping the longitude values of corresponding points on the two PTZ camera coordinate systems on the spherical coordinate system consistent, expressing visual angle difference by using the latitude difference, and respectively solving a transformation matrix for the two PTZ camera coordinate systems;

s3, selecting the master camera and the slave camera and taking the picture at the masterSelecting a tracking target on the observation image of the camera to make the main camera in any pan-tilt-zoom parameter (P)_M,T_M,Z_M) Next, an image I is observed from the camera by the tracking target_MUpper motion trackEstimating pan-tilt-zoom parameters from a cameraIn step S3, one of the two cameras is used as a master camera, the other camera is used as a slave camera, and the two cameras respectively perform visual inspection in a monitored scene according to a monitoring task, when any one of the cameras finds a suspicious object, the suspicious object is tracked by using a tracking method under a stationary camera as the master camera, and enters a master-slave working mode, and the other camera is controlled to actively track the suspicious object as the slave camera;

s4: and outputting the high-resolution panoramic image.

2. The binocular PTZ vision system master-slave tracking method of claim 1, wherein the step S1 includes calibrating an extrinsic reference matrix R and an intrinsic reference matrix K for the two PTZ cameras, respectively, the extrinsic reference matrix R being related to pan and tilt parameters only, and the axes of rotation of pan and tilt parameters intersecting perpendicularly at the camera center.

3. The binocular PTZ vision system of claim 2, wherein the step of calibrating the internal reference matrix K is:

s31: setting the pixel aspect ratio a of a camera_rAnd a gradient S;

s32: using zoom centers instead of principal points (u)₀,v₀) Ensuring pan and tilt parameters of the camera to be unchanged, obtaining an image sequence by changing zoom parameters, extracting SIFT feature points of each image, matching the feature points in two adjacent frames of images, and finally using the feature pointsObtaining a zoom center by a least square method;

s33: and constructing a variation model of the focal length f along with the zoom parameter.

4. The binocular PTZ vision system master-slave tracking method of claim 1, wherein the step S2 includes the steps of:

s41: manually acquiring N pairs of images of two cameras in the same monitoring scene, and keeping the view field of each pair of images basically consistent;

s42: extracting and matching SIFT feature points of each pair of images respectively, and estimating a basic matrix F of each pair of images_j(j ═ 1...., N), matching pairs of feature points are recordedJ is a j-th pair of images of the N pairs of images, k is a k-th pair of feature points of the j-th pair of images, and N is a positive integer;

s43: estimating the pole pair { E ] of the spherical coordinate system₁,E₂}；

S44: estimating a reference zero longitude line of the spherical coordinate system;

s45: obtaining a transformation matrix R₁,R₂。

5. The binocular PTZ vision system master-slave tracking method of claim 1, wherein the step S3 includes the steps of:

s71 selecting a target to be tracked in the observation image of the main camera;

s72: estimating parameter values of a slave camera at each moment in time

S73: and in the tracking process, calculating the distance between the central position of the tracking target and the image boundary, finishing tracking when the distance is less than a threshold value T, and restoring the preset positions of the two cameras.

6. The binocular PTZ vision system master-slave tracking method according to claim 5, wherein the step S72 includes the steps of:

s81: transforming the coordinates of the observation image of the main camera to the camera coordinates of the main camera;

s82: transforming the camera coordinates of the main camera to spherical coordinates of the main camera;

s83: mapping the spherical coordinates of the master camera to the spherical coordinates of the slave camera;

s84: transforming slave camera spherical coordinates to slave camera coordinates and estimating slave camera parameters

7. The binocular PTZ vision system master-slave tracking method of claim 1, wherein the step S4 includes the steps of:

s91: images to the main cameraCalculating background modelThe update formula at pixel (x, y) is:

wherein alpha is an updating coefficient and is set as an initial moment background modelIf it is notThen(x, y) belongs to the foreground region, otherwise, the pixel belongs to the background region, wherein th is a comparison threshold;

s92: method for capturing images of Cam-M of main camera at time t by adopting combination of feature points and direct pixelsAnd the image captured from the camera at time tCarrying out registration;

s93: the synchronized frame registration model obtained at step S92 and the main camera image obtained at step S91The background area and the foreground area of the image, calculatingA background region and a foreground region;

s94: high resolution panorama I_HIs set to be k times of the original image sizeRepresenting low resolution images of a master cameraAnd high resolution images from the cameraThe registration model of (1), then_HAndthe registration model between is:

sequentially high-resolution images from the cameraIs mapped to I_HUpdating the overlap region with an attenuation factor;

s95: sequentially high-resolution images from the cameraIs mapped to I_HIn (3), a high-resolution panorama is generated at each time.