CN113487683B - A Target Tracking System Based on Trinocular Vision - Google Patents

Abstract

The invention discloses a target tracking system based on trinocular vision. The system includes a trinocular vision module, a camera calibration module, a target position acquisition module, a target position prediction module, a virtual gun camera binocular vision calibration module, a disparity map acquisition module, Scene depth information acquisition module, PT parameter acquisition module, Z parameter acquisition module, and tracking module; a method of constructing a virtual binocular vision system by moving the gun is proposed to estimate the scene depth information in advance, and use the grounding point of the target Depth constraints are used to estimate its spatial position to achieve the purpose of uniquely determining PTZ control parameters, thereby improving the tracking accuracy.

Description

A Target Tracking System Based on Trinocular Vision

technical field

The invention relates to a target tracking system, in particular to a target tracking system based on trinocular vision.

Background technique

In some key and critical parts of military or civilian facilities, such as airport yards, oil depots, chemical plants, etc., people have higher requirements for the performance of intelligent video surveillance.

In the prior art, the method of binocular vision is usually used for target tracking, and the binocular vision system is realized by using a bolt or omnidirectional camera combined with a PTZ ball camera. First, the bolt or omnidirectional camera is used to detect moving targets, and then the PTZ ball camera Realize tracking and zoom-in capture.

Using PTZ speed dome to track moving targets is a research hotspot in recent years. The existing technology only uses the two-dimensional coordinates of the target on the image to estimate the PTZ control parameters, without considering the distance information between the target and the PTZ speed dome. When the depth changes greatly, a large tracking error will occur. Although the impact of the Z coordinate on target positioning is considered, and the depth information of the target is estimated by using information such as blanking points, these special constraints impose additional requirements on the scene content, which is only applicable to the scene content for reference. Conditions that constrain information.

To sum up, the existing target tracking methods have the problem of inaccurate tracking results.

Contents of the invention

The purpose of the present invention is to provide a target tracking system based on trinocular vision, which is used to solve the problem of inaccurate tracking results in the target tracking methods in the prior art.

In order to achieve the above tasks, the present invention adopts the following technical solutions:

A target tracking system based on trinocular vision, characterized in that the system includes a trinocular vision module, a camera calibration module, a target position acquisition module, a target position prediction module, a virtual gun camera binocular vision calibration module, and a disparity map An acquisition module, a scene depth information acquisition module, a PT parameter acquisition module, a Z parameter acquisition module and a tracking module;

The trinocular vision module is used to collect images containing moving objects. The trinocular vision module includes a bolt and a pair of PTZ ball cameras. The bolt and PTZ ball cameras are installed on slide rails. The bolt moves along the slide rail between the two PTZ ball cameras; the optical axis of the PTZ ball camera at zero position is the same as the direction of the bolt; the parameters of a pair of PTZ ball cameras same;

The trinocular vision module is used to collect images that contain the same moving target, and obtain the bolt image, the first PTZ dome image and the second PTZ dome image;

第一PTZ球机初始旋转矩阵R₀_1、第一PTZ球机平移向量t₀_1、第二PTZ球机内部参数

第二PTZ球机初始旋转矩阵R₀_2以及第二PTZ球机平移向量t₀_2；The camera calibration module is used to calibrate the internal and external parameters of the bolt and the two PTZ ball cameras, and obtain the internal parameters of the bolt, the external parameters of the bolt, and the internal parameters of the first PTZ ball camera.

Initial rotation matrix R ₀ _1 of the first PTZ ball machine, translation vector t ₀ _1 of the first PTZ ball machine, internal parameters of the second PTZ ball machine

The second PTZ ball machine initial rotation matrix R ₀ _2 and the second PTZ ball machine translation vector t ₀ _2;

The target position acquisition module is used to perform preprocessing according to the bolt image to obtain the coordinates of the target area, the coordinates of the target area are the coordinates of the target circumscribed matrix, and the coordinates of the target area include four vertices of a rectangle and Coordinates of 1 central point, m _i (u _i , v _i ), i=1,2,3,4,5;

The target position prediction module is used to perform prediction according to the target area coordinates to obtain target area predicted coordinates, the target area predicted coordinates correspond to the target area coordinates one by one, and the target area predicted coordinates Including two grounding points m ₃ and m ₄ , the two grounding points are the vertices of a rectangle, the line formed by the two grounding points is parallel to the ground and the distance between the ground and the ground is smaller than the predicted coordinates of the target area The distance between the line connecting the other two grounding points and the ground;

The binocular vision calibration module of the virtual bolt is stored with a first computer program, and the first computer program implements the following steps when executed by a processor:

Step A. Keep the position of the calibration plate fixed, control the trigger to move to the first capture point, take an image containing the calibration plate, and obtain the first calibration plate image P _A ;

Step B, controlling the bolt to be fixed on the first capture point, taking an image containing a moving target, and obtaining the first bolt image;

Step C. After controlling the action to move to the second capture point, take an image containing a moving target to obtain a second image of the action;

Step D, control the trigger to be fixed on the second snapping point, take an image containing the calibration plate, and obtain the second calibration plate image P _B, , and obtain the second calibration plate image P when the calibration plate is shot _B is in the middle of the field of view of the bolt (3) when shooting and obtaining the first calibration plate image _PA ;

Step E, using the first calibration plate image P _A and the second calibration plate image P _B to perform calibration to obtain the rotation vector R _AB and translation vector t _AB of the bolt;

A second computer program is stored in the disparity map obtaining module, and the second computer program implements the following steps when executed by the processor:

Step a, according to the rotation vector R _AB and the translation vector t _AB of the bolt (3), use a stereo correction algorithm to perform stereo correction on the first bolt image and the second bolt image, and obtain the reprojection transformation matrix Q and The rotation matrix R of the bolt; the first internal reference matrix K _A and the first projection matrix _PA of the bolt (3) when obtaining the first bolt image;

in

Step b, using formula I to map the T-point image coordinates (u _A , v _A ) of the first bolt image to the camera coordinates to obtain the T-point first camera coordinates (x′ _A , y′ _A );

Step c, using the rotation matrix R to rotate and transform the first bolt image, and using formula II to obtain the first observation coordinates (x _A , y _A ) of the point T of the bolt coordinate system;

Step d, after reprojecting the first bolt image and the second bolt image using the reprojection transformation matrix Q, using a stereo matching algorithm to obtain a disparity map;

Step e, adopt formula III to obtain the image coordinates (u' _A , v' _A ) of the T point in the disparity map;

Step f, according to the image coordinates (u' _A , v' _A ) of the T point in the disparity map, obtain the depth information of the T point in the image;

The scene depth information acquisition module is used to input the grounding points _m3 and _m4 in the bolt image as T points into the disparity map obtaining module respectively, and obtain the disparity values d3 and _d3 of the grounding points _m3 and _m4 . d ₄ ;

The scene depth information acquisition module is also used to obtain an approximate parallax value

The target position prediction module is _further used to obtain the three-dimensional coordinates X _W (5)=(X _W ( 5), Y _W (5), Z _W (5));

The PT parameter acquisition module is used to obtain the value of the P-direction rotation angle θ _{P_1} of the first PTZ dome camera and the T-direction rotation angle θ _{T_1} of the first PTZ dome camera and the P-direction rotation of the second PTZ dome camera by using Formula IV Angle θ _{P_2} and T-rotation angle θ _{T_2} of the second PTZ dome camera:

Where X _{C_1} = (X _{C_1} , Y _{C_1} , Z _{C_1} ) is the three-dimensional coordinates of the center point in the first PTZ dome camera, X _{C_1} = R _{0_1} X _W (5)+t _{0_1} ; X _{C_2} = (X _{C_2} , Y _{C_2} , Z _{C_2} ) is the three-dimensional coordinates of the center point in the second PTZ dome camera, X _{C_2} = R _{0_2} X _W (5)+t _{0_2} ;

A third computer program is stored in the Z parameter acquisition module, and the third computer program implements the following steps when executed by the processor:

Step 1. Use Formula V to obtain the first PTZ dome rotation matrix R _PT _1 and the second PTZ dome rotation matrix R _PT _2;

Step 2. Obtain the coordinates (-X _max , Y _max ), (X _max , Y _max ), (X _max ,-Y _max ) and ( _-Xmax , _-Ymax );

Step 3. If the aspect ratio of the rectangle is greater than or equal to the aspect ratio of the dome camera, set the X axis as the main direction, otherwise the Y axis is the main direction;

或Y轴方向上的焦距

Step 4. Use Formula VI to obtain the focal length in the X-axis direction

or the focal length in the Y-axis direction

Among them, k is a proportional coefficient, k is a constant, Z _E represents the zoom control parameter of the dome camera, Z _E is a constant, W ₁ represents the resolution length of the first PTZ dome camera, W ₁ is a constant, W ₂ represents the second PTZ The length of the resolution of the ball camera, W ₂ is a constant; H ₁ represents the resolution width of the first PTZ ball camera, H ₁ is a constant, H ₂ represents the resolution width of the second PTZ ball camera, H ₂ is a constant ;

Step 5. When the main direction is set as the X axis in step 3, use Newton's method to solve formula VII to obtain the control parameter Z:

When the main direction is set as the Y axis in step 3, use Newton's method to solve formula VIII to obtain the focal length control parameter value Z of the PTZ ball camera:

The f _{x_1} (Z), f _{x_2} (Z), f _{y_1} (Z) and f _{y_2} (Z) are fitting functions obtained by calibration of the camera calibration module;

The tracking module is used to use the value of the P-direction rotation angle θ _{P_1} of the first PTZ dome camera and the T-direction rotation angle θ _{T_1} of the first PTZ dome camera obtained by the PT parameter acquisition module, and the P-direction rotation angle of the second PTZ dome camera. The rotation angle θ _{P_2} and the T-direction rotation angle θ _{T_2} of the second PTZ ball camera control the PT angles of the first PTZ ball camera and the second PTZ ball camera;

It is used to use the Z parameter acquisition module to respectively obtain the focal length control parameter values of the first PTZ dome camera and the second PTZ dome camera, control the focal length parameters of the first PTZ dome camera and the second PTZ dome camera, and complete the tracking.

Further, the stereo correction algorithm in step a of the disparity map obtaining module is a Bouguet stereo correction algorithm.

Further, both the camera calibration module and the virtual camera binocular vision calibration module use Zhang Zhengyou's calibration algorithm for calibration.

Further, the target position acquisition module uses the DACB foreground detection algorithm to process the bolt image, and after obtaining the foreground area containing shadows and moving objects, uses the shadow elimination algorithm to obtain the coordinates of the target area.

Further, the target position prediction module is used to predict the coordinates of the target area by using the Kalman prediction algorithm to obtain the predicted coordinates of the target area.

Compared with the prior art, the present invention has the following technical effects:

1. The target tracking system based on trinocular vision provided by the present invention proposes a method of constructing a virtual binocular vision system by moving the gun to estimate the depth information of the scene in advance, and estimate it based on the depth constraint of the target's grounding point Its spatial position achieves the purpose of uniquely determining PTZ control parameters, thereby improving the accuracy of target tracking;

2. The target tracking system based on trinocular vision provided by the present invention realizes the acquisition of scene depth under the virtual binocular vision system, and obtains the coordinate correspondence between real-time images and disparity maps, which improves the accuracy of scene depth acquisition, thereby Improved target tracking accuracy;

3. The target tracking system based on trinocular vision provided by the present invention makes full use of the stable parameters of the bolt as the main camera, and combines the scene depth information obtained in the system initialization stage to realize the three-dimensional coordinate estimation of the moving target, and thereby realize The calculation of PTZ control parameters achieves the purpose of accurate calculation;

4. The target tracking system based on trinocular vision provided by the present invention calculates the tracking control parameters of the PTZ ball camera by using the scene depth information and the predicted target position ground point, thereby improving the accuracy and convenience of the control parameter calculation.

Description of drawings

Fig. 1 is the structural representation of the trinocular vision module provided by the present invention;

Fig. 2 is a schematic diagram of the movement path of the bolt provided on the slide rail in one embodiment of the present invention;

Fig. 3 is a schematic flow chart of the PTZ control parameter calculation provided in one embodiment of the present invention;

Fig. 4 is a schematic diagram of a parallel binocular vision system provided in an embodiment of the present invention;

5 is a schematic diagram of the geometric relationship between the predicted target position and the PT angle provided in one embodiment of the present invention;

FIG. 6 is a schematic diagram illustrating a moving target in a gun image provided in an embodiment of the present invention;

Fig. 7 is a graph of the angle fitting results in the P direction in the least squares fitting of the PT control parameters and the actual rotation angle provided in an embodiment of the present invention;

Fig. 8 is a graph of the angle fitting results in the T direction in the least squares fitting between the PT control parameters and the actual rotation angle provided in an embodiment of the present invention;

Fig. 9 is a schematic diagram of the target position after the PT of the PTZ dome machine is rotated according to an embodiment of the present invention;

Fig. 10 is a schematic diagram of reconstruction of a target rectangle in a coordinate system provided in an embodiment of the present invention.

The numbers in the figure represent: 1-rail, 2-PTZ ball camera, 3-bolt.

Detailed ways

The present invention will be described in detail below in conjunction with the accompanying drawings and embodiments. So that those skilled in the art can better understand the present invention. It should be noted that in the following description, when detailed descriptions of known functions and designs may dilute the main content of the present invention, these descriptions will be omitted here.

The definitions or conceptual connotations involved in the present invention are described below:

Bolt: Bolt is a kind of surveillance camera. The bolt looks like a cuboid, with a C/CS lens mount on the front.

PTZ dome camera: Pan-Tilt-Zoom dome camera, shorthand for Pan/Tilt/Zoom in security monitoring applications, represents a dome surveillance camera with omni-directional (left/right/up and down) movement of the pan/tilt and lens zoom and zoom control.

Example

In the embodiment, a target tracking system based on trinocular vision is disclosed. The system includes a trinocular vision module, a camera calibration module, a target position acquisition module, a target position prediction module, a virtual gun machine binocular vision calibration module, and a disparity map acquisition Module, scene depth information acquisition module, PT parameter acquisition module, Z parameter acquisition module and tracking module;

As shown in Figure 1, the trinocular vision module provided by the present invention is used to collect images containing moving objects, the trinocular vision module includes a bolt 3 and two PTZ ball cameras 2, the bolt 3 and the PTZ ball machine 2 are installed on the slide rail 1, the slide rail 1 is arranged parallel to the horizontal plane, and the bolt 3 moves along the slide rail 1 in the middle of the two PTZ ball machines 2; the The optical axis of PTZ ball camera 2 at zero position is in the same direction as bolt camera 3; the parameters of a pair of PTZ ball cameras are exactly the same;

The present invention utilizes the constant characteristic of the internal parameters of the bolt, as shown in Figure 1, by fixing it at two different positions on the guide rail 1 to realize a virtual binocular vision system, and simultaneously obtain the calibration template images of the two positions and scene image. FIG. 2 is a schematic diagram of the moving path of the bolt 3 on the guide rail 1 . The positional relationship between the B-position bolt relative to the A-position bolt is described by a rotation vector r _AB and a translation vector t _AB .

The trinocular vision module is used to collect images that contain the same moving target, and obtain the bolt image, the first PTZ dome image and the second PTZ dome image;

第一PTZ球机初始旋转矩阵R₀_1、第一PTZ球机平移向量t₀_1、第二PTZ球机内部参数

第二PTZ球机初始旋转矩阵R₀_2以及第二PTZ球机平移向量t₀_2；The camera calibration module is used to calibrate the internal and external parameters of the bolt and the two PTZ ball cameras, and obtain the internal parameters of the bolt, the external parameters of the bolt, and the internal parameters of the first PTZ ball camera.

Initial rotation matrix R ₀ _1 of the first PTZ ball machine, translation vector t ₀ _1 of the first PTZ ball machine, internal parameters of the second PTZ ball machine

The second PTZ ball machine initial rotation matrix R ₀ _2 and the second PTZ ball machine translation vector t ₀ _2;

The PTZ parameter acquisition method provided by the present invention is shown in FIG. 3 .

The target position acquisition module is used to perform preprocessing according to the bolt image to obtain the coordinates of the target area, the coordinates of the target area are the coordinates of the target circumscribed matrix, and the coordinates of the target area include four vertices of a rectangle and Coordinates of 1 central point, m _i (u _i , v _i ), i=1,2,3,4,5;

In this embodiment, to obtain the precise control parameters of the PTZ parameters at the predicted position of the target, and to control the motion of the PTZ dome camera so that its optical axis is facing the center of the target, it is necessary to obtain the three-dimensional coordinates of the target in the PTZ dome camera coordinate system through coordinate transformation.

Optionally, the target position acquisition module uses the DACB foreground detection algorithm to process the bolt image, obtains the foreground area containing shadows and moving targets, and then uses the shadow elimination algorithm to obtain the coordinates of the target area.

In this embodiment, the DACB foreground detection algorithm first obtains the foreground area including shadows and moving objects, and then removes the shadow components by the shadow elimination algorithm to obtain the area description of the moving object and obtain the coordinates of the object area.

The target position prediction module is used to perform prediction according to the target area coordinates to obtain target area predicted coordinates, the target area predicted coordinates correspond to the target area coordinates one by one, and the target area predicted coordinates Including two grounding points m ₃ and m ₄ , the two grounding points are the vertices of a rectangle, the line formed by the two grounding points is parallel to the ground and the distance between the ground and the ground is smaller than the predicted coordinates of the target area The distance between the line connecting the other two grounding points and the ground;

Optionally, the target position prediction module is used to predict the coordinates of the target area by using the Kalman prediction algorithm to obtain the predicted coordinates of the target area.

Theoretically, the image position of the target in the current frame is obtained through the background modeling algorithm, combined with the Kalman prediction algorithm, the predicted position of the target after ΔT time can be obtained, as shown in the following formula.

Among them, d _x and d _y are the prediction of the moving speed of the object in the X and Y directions given by the Kalman prediction algorithm based on historical information, and ΔT is the sum of a series of delays such as video delay and control delay, as shown in the following formula.

ΔT＝T _QJ +ΔT _VISCA +ΔT _PT

ΔT _PT is the time required for the PTZ ball camera to rotate the optical axis from the current PT angle to the target center, the larger the rotation angle, the greater the ΔT _PT . The rotation angle is affected by the total delay time ΔT, that is, the movement of the target within the ΔT delay time will affect the tracking accuracy of the PTZ dome camera. It can be seen that the PT rotation angle and the delay time are mutually influenced and restricted by each other, so it is difficult to obtain the exact solution of the two parameters at the same time. For this reason, the present invention has done following processing to this problem:

(1) Use the current position of the target instead of the predicted position to calculate the angle required for PZ control to obtain the PT delay;

(2) Put the PT delay into the above formula to obtain the total delay ΔT, and use it to further calculate various control parameters.

Using this similar iterative processing method, the approximate solution of various control parameters can be quickly obtained, and the tracking control of the PTZ ball machine can be implemented.

In this embodiment, the position prediction of the target after a certain delay can be obtained by using the Kalman prediction algorithm combined with the histogram statistical information. And the result is represented by a rectangle, as shown in Figures 5 and 6, the number 3 in Figure 5 represents the trigger, and the number 2 represents the PTZ ball camera, and the center point C of the rectangle is the center position of the moving target, and these five points are set at The coordinates in the bolt image are m _i (u _i , v _i ), i=1, 2, 3, 4, 5. Among them, the two points 3 and 4 can be regarded as the contact point between the target moving to the prediction rectangle and the ground after ΔT time, which is called the grounding point. In this embodiment, the three-dimensional coordinates of the four vertices and the center point C of the target rectangle will be approximately estimated by means of these two grounding points.

The binocular vision calibration module of the virtual bolt is stored with a first computer program, and the first computer program implements the following steps when executed by a processor:

Step A, keep the position of the calibration plate fixed, control the trigger 3 to move to the first capture point, take an image containing the calibration plate, and obtain the first calibration plate image _PA ;

Step B, controlling the trigger 3 to be fixed on the first capture point, shooting an image containing a moving target, and obtaining the first trigger image;

Step C, controlling the trigger 3 to move to the second capture point and then shooting an image containing a moving target to obtain a second trigger image;

Step D, control the trigger 3 to be fixed on the second snapping point, take an image containing the calibration plate, and obtain the second calibration plate image P _B, , the calibration plate is captured to obtain the second calibration plate image P _B is in the middle of the field of view of the bolt 3 when shooting and obtaining the first calibration plate image _PA ;

Step E, using the first calibration plate image P _A and the second calibration plate image P _B to perform calibration to obtain the rotation vector R _AB and translation vector t _AB of the bolt 3;

In order to simplify operations and reduce errors, the present invention uses the following four steps to achieve calibration and capture of scene images:

(1) Fix the gun at point B, capture a scene image S _B , and save it;

(2) Keep the position of the bolt unchanged, and place a calibration template fixedly in the middle of the bolt’s field of view slightly to the left (to ensure that the template can still be located in the middle of the bolt’s field of view after the bolt is moved to point A), and take a snapshot A calibration template image P _B ;

(3) Keep the position of the calibration template unchanged, move the trigger to point A and securely fix it (as the final working position), and capture the second calibration template image P _A ;

(4) Remove the calibration template, and capture the second scene image S _A .

Different from the binocular vision system in the usual sense, the binocular in the binocular system provided by the present invention is actually the same camera located at two different positions, so its internal parameters are completely consistent and precisely known, so only calibration between them is required. mutual positional relationship. Therefore, only using two calibration template images can also achieve better calibration results, and obtain r _AB and t _AB with sufficient precision. Here, the calibration uses the GML MatLab Camera Calibration Toolbox toolbox.

Optionally, in Step E, Zhang Zhengyou's calibration algorithm is used for calibration.

In this implementation, the trigger 3 is calibrated, and the calibration results are the rotation vector r _AB and the translation vector t _AB after the Rodriguez transformation, and the calibration result is:

It should be noted that all element values of the r _AB vector and the t _y and t _z components in t _AB should be all 0 in theory, but the actual vector elements obtained are not 0 but three smaller values, which is It shows that there is a slight deviation in the movement process. The smaller their values are, the less the projection correction of the image is during stereo correction, and the higher the accuracy of the finally obtained scene depth information is.

A second computer program is stored in the disparity map obtaining module, and the second computer program implements the following steps when executed by the processor:

Step a, according to the rotation vector R _AB and the translation vector t _AB of the bolt 3, use a stereo correction algorithm to perform stereo correction on the first bolt image and the second bolt image, and obtain the reprojection transformation matrix Q and the bolt The rotation matrix R; obtain the first internal reference matrix K _A and the first projection matrix _PA of the bolt (3) when acquiring the first bolt image;

in

In the present invention, as shown in Figures 2 and 4, since the camera at point A and the camera at point B are the same camera, that is, the two cameras in the binocular camera are exactly the same, so the internal reference matrix, projection matrix, etc. are also completely identical. Same, so the disparity map can be obtained with only the left camera without multiple calculations. Or use the same method to obtain the second internal reference matrix K _B and the second projection matrix P _B for the camera at point B (the right camera in the binocular), or only use the right camera to obtain the disparity map.

Therefore, the first internal reference matrix K _A or the second internal reference matrix K B obtained at point A or point _B , the first projection matrix P _A or the second projection matrix P _B are as follows:

Optionally, the stereo correction algorithm in step a is a Bouguet stereo correction algorithm.

In this embodiment, after obtaining R _AB and t _AB , use the Bouguet stereo correction algorithm to perform stereo correction on the scene images _SA and S _B to obtain the projection matrix required for correction (since the distortion correction has been completed in advance, the correction Regardless of the distortion parameters, all elements of the distortion vector have values of 0). The row alignment operation of the images can be completed by reprojecting the two images, and then the disparity map can be obtained through the stereo matching algorithm.

In this embodiment, the corrected left and right camera matrices K _A and K _B , and the projection matrices _PA and P _B are as follows, where the trigger 3 at point A is the left camera in the binocular camera, and point B is Bolt 3 above is the right camera in the binocular camera:

(wherein, α _A =α _B =0, both α _A and α _B are the ratio of pixel distortion, due to the improvement of the production process, the parameters of cameras currently on the market can be considered as 0).

The projection matrix can convert the 3D points in the homogeneous coordinates to the 2D points in the homogeneous coordinate system, and the screen coordinates can be obtained as (x/w, y/w). If the screen coordinates and the camera internal parameter matrix are given, the two-dimensional point can be reprojected to obtain the three-dimensional coordinates, and the reprojection matrix Q is as follows:.

All the parameters except c′ _x in the formula come from the first bolt image, and c′ _x is the x-coordinate of the principal point on the second bolt image. If the chief ray intersects at infinity, then c' _x = c _x , and the bottom right term is 0.

Given a two-dimensional homogeneous coordinate and the corresponding disparity d, the following formula can be used to project this point into a three-dimensional coordinate system to obtain its space coordinates (X/W, Y/W, Z/W), which include Depth information of spatial points.

Through the above-mentioned Bouguet correction algorithm, the transformation matrices required for re-projection can be obtained, as shown in the following formula, to realize the stereo correction of the image pair, and virtualize a parallel binocular stereo vision system as shown in Figure 4, which can be used for the follow-up along the epipolar line. Match search and deep search lay the foundation.

In this embodiment, since the algorithm needs to perform epipolar correction on the two images during the stereo matching process, the image coordinates between the obtained scene disparity map and the real-time image of the bolt are not one-to-one correspondence, but there is a mapping transformation relation. In the present invention, the camera at point A is regarded as the left camera in the binocular, and the internal reference matrix of the camera is K _QJ , the distortion vector is d _QJ , and the Bouguet stereo correction algorithm obtains the first rotation matrix R _A and the first projection matrix P _A . The bolt image has been corrected for distortion in advance, so the d _QJ here is filled with all 0s.

Step b, using formula I to map the T-point image coordinates (u _A , v _A ) of the first bolt image to the camera coordinates to obtain the T-point first camera coordinates (x′ _A , y′ _A );

In this embodiment, the first camera coordinates (x′ _A , y′ _A ) at point T and the second camera coordinates (x′ _B , y′ _B ) at point T are obtained as follows:

Step c, using the rotation matrix R to rotate and transform the first bolt image, and using formula II to obtain the first observation coordinates (x _A , y _A ) of the point T of the bolt coordinate system;

In this embodiment, the first observed coordinates (x _A , y _A ) of point T and the second observed coordinates (x _B , y _B ) of point T of the bolt coordinate system are obtained:

Step d, after reprojecting the first bolt image and the second bolt image using the reprojection transformation matrix Q, using a stereo matching algorithm to obtain a disparity map;

Step e, adopt formula III to obtain the image coordinates (u' _A , v' _A ) of the T point in the disparity map;

In this embodiment, the image coordinates (u′ _A , v′ _A ) and (u′ _B , v′ _B ) of point T in the disparity map are obtained:

Step f, according to the image coordinates (u' _A , v' _A ) of the T point in the disparity map, obtain the depth information of the T point in the image;

将此点投影到三维坐标系中，得到其空间坐标(X/W,Y/W,Z/W)，该坐标包含了空间点的深度信息。In this embodiment, the depth information of point T in the image can be obtained according to the image coordinates (u′ _A , v′ _A ) or (u′ _B , v′ _B ) of the disparity map, given a two-dimensional homogeneous coordinate And the corresponding disparity d, then available

Project this point into a three-dimensional coordinate system to obtain its space coordinates (X/W, Y/W, Z/W), which contain the depth information of the space point.

The scene depth information acquisition module is used to input the grounding points _m3 and _m4 in the bolt image as T points into the disparity map obtaining module respectively, and obtain the disparity values d3 and _d3 of the grounding points _m3 and _m4 . d ₄ ;

The scene depth information acquisition module is also used to obtain an approximate parallax value

In this embodiment, first obtain the coordinate positions _{m d_3 (} u _{d_3} , v _{d_3} ) and m _{d_4} (u _{d_4} , v _{d_4} ) of the grounding points m 3 and _m 4 in the disparity map, and obtain them from the scene disparity map The parallax values of the two points are recorded as d ₃ and d ₄ , and their average value d ₀ is taken as the approximate parallax value of the four vertices of the entire target rectangle and the center point C, and then the four vertices and The three-dimensional coordinates X _W (i)=(X _W (i), Y _W (i), Z _W (i))i=1,2,3,4,5 of point C in the bolt coordinate system.

The target position prediction module is _further used to obtain the three-dimensional coordinates X _W (5)=(X _W ( 5), Y _W (5), Z _W (5));

The PT parameter acquisition module is used to obtain the P-direction rotation angle θ _{P_1} of the first PTZ dome camera and the value θ _{T_1} of the T-direction rotation angle of the first PTZ dome camera and the P-direction rotation of the second PTZ dome camera by using Formula I Angle θ _{P_2} and T-rotation angle θ _{T_2} of the second PTZ dome camera:

Where X _{C_1} = (X _{C_1} , Y _{C_1} , Z _{C_1} ) is the three-dimensional coordinates of the center point in the first PTZ dome camera, X _{C_1} = R _{0_1} X _W (5)+t _{0_1} ; X _{C_2} = (X _{C_2} , Y _{C_2} , Z _{C_2} ) is the three-dimensional coordinates of the center point in the second PTZ dome camera, X _{C_2} = R _{0_2} X _W (5)+t _{0_2} ;

In this embodiment, using the initial relationship R _{0_s} and t _{0_s} (s=1, 2 respectively representing two PTZ ball cameras) between the gun camera coordinate system and the coordinates of the two PTZ ball cameras, calculate point C at the first PTZ ball camera The coordinates in the camera coordinate system of the PTZ ball camera.

In this embodiment, a method for solving the actual PT parameters is also provided, which is obtained by the following formula:

According to the fitting formula of PT parameters, use the formula to calculate the control parameters actually required by PT.

In order to improve the control accuracy of the PTZ dome camera during active tracking, the present invention proposes a time-weighted PT control parameter least square fitting algorithm.

The present invention adopts formula mathematical model between PT control parameter and angle:

After the PTZ dome camera works for a period of time, the corresponding data of N groups of θ and θ' can be obtained. According to the least square method, the optimal solution of parameters a and b in the sense of least square can be obtained, as shown in the following formula.

For PT control parameters, recent data trends can better reflect the current state of the PT control system and provide more useful information for the future, while early data are less useful. Therefore, in this embodiment, the weights are set according to the time sequence of the data, and the weights adopt the exponential weighting method. In order to prevent the problem of excessive data volume caused by long-term operation, this embodiment sets a data validity period, that is, only the latest N sets of data are kept, and the least square calculation is not performed on expired data. At the same time, in order to prevent the interference of wild points, the present invention will judge the data with too large error (that is, the data pair with large deviation between the control parameter value and the accurate value obtained by actual solution) and ignore it. The present invention takes N=20, and according to the value of N, the difference between 1 and the sum of weights is superimposed on the latest set of data to ensure that the sum of weights is 1.

如下式。表1为N取20时各组数据的权值计算结果(取s＝0.2，有w_Rest＝0.011529)，使用权值后的参数求解公式如式所示。The specific method is as follows: first assume that the weight of the latest data (Nth group) is w _N =s (0<s<1), and the weight of the tth group of data is w _t =s(1-s) ^Nt . Since the sum of these N weights is less than 1, the present invention divides the remaining weights into N equal parts and superimposes them on the existing N weights to obtain normalized N sets of data weights

as follows. Table 1 shows the weight calculation results of each group of data when N is 20 (take s = 0.2, w _Rest = 0.011529), and the parameter solution formula after using the weight is shown in the formula.

Table 1 Weight distribution when N=20 (s=0.2)

Figures 7 and 8 show the weighted least squares parameter fitting results after 20 tracking experiments of the left PTZ ball camera. Figure 7 shows the angle fitting results in the P direction, and Figure 8 shows the angle fitting results in the T direction. The following formula is the obtained fitting function. It should be noted that the parameters of the fitting function will continue to change as the number of tracking increases and the fitting data is updated. This mechanism of updating the fitting function online can effectively adapt to the constantly changing rotation angle error of the PTZ ball machine, ensuring the accuracy and sustainability of the fitting of the control parameters.

The first computer program is stored in the Z parameter acquisition module, and the first computer program implements the following steps when executed by the processor:

Step a, using Formula V to obtain the first PTZ dome rotation matrix R _PT _1 and the second PTZ dome rotation matrix R _PT _2;

Step b. Obtain the coordinates (-X _max , Y _max ), (X _max , Y _max ), (X _max ,-Y _max ) and ( _-Xmax , _-Ymax );

Step c. If the aspect ratio of the rectangle is greater than or equal to the aspect ratio of the ball machine, set the X axis as the main direction, otherwise the Y axis is the main direction;

或Y轴方向上的焦距

Step d, using formula VI to obtain the focal length in the X-axis direction

or the focal length in the Y-axis direction

Among them, k is a proportional coefficient, k is a constant, Z _E represents the zoom control parameter of the dome camera, Z _E is a constant, W ₁ represents the resolution length of the first PTZ dome camera, W ₁ is a constant, W ₂ represents the second PTZ The length of the resolution of the ball camera, W ₂ is a constant; H ₁ represents the resolution width of the first PTZ ball camera, H ₁ is a constant, H ₂ represents the resolution width of the second PTZ ball camera, H ₂ is a constant ;

Step e, when the main direction set in step c is the X axis, Newton's method is used to solve formula VII to obtain the control parameter Z:

When the main direction set in step c is the Y axis, Newton's method is used to solve formula VIII to obtain the control parameter Z:

The f _{x_1} (Z), f _{x_2} (Z), f _{y_1} (Z) and f _{y_2} (Z) are fitting functions calibrated by the camera calibration module.

In this embodiment, in order to calculate the Z control parameter, using the obtained PT control parameter, the calculation process of the Z parameter will be introduced in the following five steps. Since the calculation process of the control parameters of the two PTZ ball machines is the same, the difference between PTZ ball machines 1 and 2 is not considered here.

(1) Calculate the three-dimensional coordinates of the target in the PTZ dome camera coordinate system after the PT movement. The process of rotating and transforming the four vertices of the target rectangle in the dome coordinate system to obtain its three-dimensional coordinates in the PTZ dome coordinate system.

(2) Target rectangle reconstruction

After obtaining the PT tracking result shown in Figure 9, it can be considered that the optical axis of the PTZ ball camera passes through the center of the target, and the originally defined rectangle is no longer a rectangle due to the influence of projection (here do not consider the position of each point in Z axis coordinates), such as P′ ₁ P′ ₂ P′ ₃ P′ ₄ in Figure 10. Therefore, the present invention selects the maximum coordinate values Xmax and Ymax in the two directions of X and Y, such as formula, and uses them as parameters to reconstruct the rectangle P″ ₁ P″ ₂ P″ ₃ P″ ₄ . After reconstruction, the X and Y coordinates of the four vertices of P″ ₁ P″ ₂ P″ ₃ P″ ₄ are (-Xmax, Ymax), (Xmax, Ymax), (Xmax, -Ymax) and (-Xmax ,-Ymax).

(3) Main direction selection

In this embodiment, the imaging resolution of the PTZ ball camera is 704×576, that is, there is an aspect ratio q=W/H=704/576=1.222. Define X as the main direction when the aspect ratio in the rectangle is higher than q, otherwise Y is the main direction, as shown in the following formula.

(4) Theoretical focal length solution

In this embodiment, it is expected that the final imaging size occupies an appropriate proportion in the entire PTZ dome camera image, and the proportion is determined by the length or width in the main direction, as shown in FIG. 10 . Let the ratio be k (k<1), so after determining the main direction, the ratio of the imaging size in the main direction to the image size of the PTZ dome camera can be calculated according to the triangular geometric relationship in the pinhole imaging model. The best focal length, the formula is the focal length calculation process when the main direction is X.

(5) Z control parameter solution

According to the f _x (or f _y ) fitting function f _x (Z) obtained when the PTZ ball machine is calibrated, construct the equation, and use Newton's method to solve the Z value when the equation is true, and this value is the PTZ ball machine Focal length control parameter value. When the main direction is Y, the focal length in steps (4) and (5) is replaced by fy, and W=704 in the formula is replaced by H=576.

In this embodiment, the flexible system architecture of the trinocular vision system is used to estimate the depth of the scene, and the tracking control parameters of the PTZ ball camera are obtained in combination with the predicted position of the moving target, so as to improve the accuracy of the acquisition of the PTZ control parameters.

The tracking module is used to use the value of the P-direction rotation angle θ _{P_1} of the first PTZ dome camera and the T-direction rotation angle θ _{T_1} of the first PTZ dome camera obtained by the PT parameter acquisition module, and the P-direction rotation angle θ of the second PTZ dome camera _{P_2} and the T-rotation angle θ of the second PTZ ball camera _{T_2} controls the PT angle of the first PTZ ball camera and the second PTZ ball camera;

It is used to use the Z parameter acquisition module to respectively obtain the focal length control parameter values of the first PTZ dome camera and the second PTZ dome camera, control the focal length parameters of the first PTZ dome camera and the second PTZ dome camera, and complete the tracking.

Through the description of the above embodiments, those skilled in the art can clearly understand that the present invention can be realized by means of software plus necessary general-purpose hardware, and of course also by hardware, but in many cases the former is a better embodiment . Based on this understanding, the essence of the technical solution of the present invention or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product is stored in a readable storage medium, such as a floppy disk of a computer , a hard disk or an optical disk, etc., including several instructions to enable a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of various embodiments of the present invention.

Claims (5)

1. A target tracking system based on trinocular vision, characterized in that, said system comprises a trinocular vision module, a camera calibration module, a target position acquisition module, a target position prediction module, a virtual gun camera binocular vision calibration module, A disparity map acquisition module, a scene depth information acquisition module, a PT parameter acquisition module, a Z parameter acquisition module and a tracking module; Described trinocular vision module is used for collecting the image that contains moving object, and described trinocular vision module comprises a bolt (3) and a pair of PTZ dome camera (2), described bolt (3) and The PTZ ball camera (2) is installed on the slide rail (1), and the slide rail (1) is arranged parallel to the horizontal plane, and the bolt (3) is on the slide rail (1) along the A pair of PTZ ball cameras (2) move in the middle; the optical axis of the PTZ ball cameras (2) at zero position is in the same direction as the bolt (3); the parameters of the pair of PTZ ball cameras (2) are the same ; The trinocular vision module is used to collect images that contain the same moving target, and obtain the bolt image, the first PTZ dome image and the second PTZ dome image; The camera calibration module is used to calibrate the internal and external parameters of the bolt and the two PTZ ball cameras, and obtain the internal parameters of the bolt, the external parameters of the bolt, and the internal parameters of the first PTZ ball camera.

Initial rotation matrix R ₀ _1 of the first PTZ ball machine, translation vector t ₀ _1 of the first PTZ ball machine, internal parameters of the second PTZ ball machine

The second PTZ ball machine initial rotation matrix R ₀ _2 and the second PTZ ball machine translation vector t ₀ _2; The target position acquisition module is used to perform preprocessing according to the bolt image to obtain the coordinates of the target area, the coordinates of the target area are the coordinates of the target circumscribed matrix, and the coordinates of the target area include four vertices of a rectangle and Coordinates of 1 central point, m _i (u _i , v _i ), i=1,2,3,4,5; The target position prediction module is used to perform prediction according to the target area coordinates to obtain target area predicted coordinates, the target area predicted coordinates correspond to the target area coordinates one by one, and the target area predicted coordinates Including two grounding points m ₃ and m ₄ , the two grounding points are the vertices of a rectangle, the line formed by the two grounding points is parallel to the ground and the distance between the ground and the ground is smaller than the predicted coordinates of the target area The distance between the line connecting the other two grounding points and the ground; The binocular vision calibration module of the virtual bolt is stored with a first computer program, and the first computer program implements the following steps when executed by a processor: Step A, keep the position of the calibration plate fixed, control the trigger (3) to move to the first snapshot point, take a pair of images containing the calibration plate, and obtain the first calibration plate image _PA ; Step B, controlling the bolt (3) to be fixed on the first capture point, taking an image containing a moving target, and obtaining the first bolt image; Step C, controlling the trigger (3) to move to the second capture point and take an image containing a moving target to obtain a second trigger image; Step D, control the gunlock (3) to be fixed on the second snapping point, take an image containing the calibration plate, and obtain the second calibration plate image P _B, , and the calibration plate obtains the second calibration plate when shooting The board image P _B is all in the middle of the field of view of the bolt (3) when the first calibration board image P _A is obtained by shooting; Step E, using the first calibration plate image P _A and the second calibration plate image P _B to perform calibration to obtain the rotation vector R _AB and translation vector t _AB of the bolt (3); A second computer program is stored in the disparity map obtaining module, and the second computer program implements the following steps when executed by the processor: Step a, according to the rotation vector R _AB and the translation vector t _AB of the bolt (3), use a stereo correction algorithm to perform stereo correction on the first bolt image and the second bolt image, and obtain the reprojection transformation matrix Q and The rotation matrix R of the bolt; the first internal reference matrix K _A and the first projection matrix _PA of the bolt (3) when obtaining the first bolt image; in

Step b, using formula I to map the T-point image coordinates (u _A , v _A ) of the first bolt image to the camera coordinates to obtain the T-point first camera coordinates (x′ _A , y′ _A );

Step c, using the rotation matrix R to rotate and transform the first bolt image, and using formula II to obtain the first observation coordinates (x _A , y _A ) of the point T of the bolt coordinate system;

Step d, after reprojecting the first bolt image and the second bolt image using the reprojection transformation matrix Q, using a stereo matching algorithm to obtain a disparity map; Step e, adopt formula III to obtain the image coordinates (u' _A , v' _A ) of the T point in the disparity map;

Step f, according to the image coordinates (u' _A , v' _A ) of the T point in the disparity map, obtain the depth information of the T point in the image; The scene depth information acquisition module is used to input the grounding points _m3 and _m4 in the bolt image as T points into the disparity map obtaining module respectively, and obtain the disparity values d3 and _d3 of the grounding points _m3 and _m4 . d ₄ ; The scene depth information acquisition module is also used to obtain an approximate parallax value

The target position prediction module is _further used to obtain the three-dimensional coordinates X _W (5)=(X _W ( 5), Y _W (5), Z _W (5)); The PT parameter acquisition module is used to obtain the value of the P-direction rotation angle θ _{P_1} of the first PTZ dome camera and the T-direction rotation angle θ _{T_1} of the first PTZ dome camera and the P-direction rotation of the second PTZ dome camera by using Formula IV Angle θ _{P_2} and T-rotation angle θ _{T_2} of the second PTZ dome camera:

Where X _{C_1} = (X _{C_1} , Y _{C_1} , Z _{C_1} ) is the three-dimensional coordinates of the center point in the first PTZ dome camera, X _{C_1} = R _{0_1} X _W (5)+t _{0_1} ; X _{C_2} = (X _{C_2} , Y _{C_2} , Z _{C_2} ) is the three-dimensional coordinates of the center point in the second PTZ dome camera, X _{C_2} = R _{0_2} X _W (5)+t _{0_2} ; A third computer program is stored in the Z parameter acquisition module, and the third computer program implements the following steps when executed by the processor: Step 1. Use Formula V to obtain the first PTZ dome rotation matrix R _PT _1 and the second PTZ dome rotation matrix R _PT _2;

Step 2. Obtain the coordinates (-X _max , Y _max ), (X _max , Y _max ), (X _max ,-Y _max ) and ( _-Xmax , _-Ymax ); Step 3. If the aspect ratio of the rectangle is greater than or equal to the aspect ratio of the dome camera, set the X axis as the main direction, otherwise the Y axis is the main direction; Step 4. Use Formula VI to obtain the focal length in the X-axis direction

or the focal length in the Y-axis direction

Among them, k is a proportional coefficient, k is a constant, Z _E represents the zoom control parameter of the dome camera, Z _E is a constant, W ₁ represents the resolution length of the first PTZ dome camera, W ₁ is a constant, W ₂ represents the second PTZ The length of the resolution of the ball camera, W ₂ is a constant; H ₁ represents the resolution width of the first PTZ ball camera, H ₁ is a constant, H ₂ represents the resolution width of the second PTZ ball camera, H ₂ is a constant ; Step 5. When the main direction is set as the X axis in step 3, use Newton's method to solve formula VII to obtain the control parameter Z:

When the main direction is set as the Y axis in step 3, use Newton's method to solve formula VIII to obtain the focal length control parameter value Z of the PTZ ball camera:

The f _{x_1} (Z), f _{x_2} (Z), f _{y_1} (Z) and f _{y_2} (Z) are fitting functions obtained by calibration of the camera calibration module; The tracking module is used to use the value of the P-direction rotation angle θ _{P_1} of the first PTZ dome camera and the T-direction rotation angle θ _{T_1} of the first PTZ dome camera obtained by the PT parameter acquisition module, and the P-direction rotation angle of the second PTZ dome camera. The rotation angle θ _{P_2} and the T-direction rotation angle θ _{T_2} of the second PTZ ball camera control the PT angles of the first PTZ ball camera and the second PTZ ball camera; It is used to use the Z parameter acquisition module to respectively obtain the focal length control parameter values of the first PTZ dome camera and the second PTZ dome camera, control the focal length parameters of the first PTZ dome camera and the second PTZ dome camera, and complete the tracking. 2. The target tracking system based on trinocular vision as claimed in claim 1, wherein the stereo correction algorithm in the step a of the disparity map obtaining module is a Bouguet stereo correction algorithm. 3. The target tracking system based on trinocular vision as claimed in claim 1, characterized in that, Zhang Zhengyou's calibration algorithm is used for calibration in the camera calibration module and the binocular vision calibration module of the virtual gun. 4. the target tracking system based on trinocular vision as claimed in claim 1, is characterized in that, described target position acquisition module adopts DACB foreground detection algorithm to process bolt image, obtains the foreground area that comprises shadow and moving target Finally, use the shadow elimination algorithm to obtain the coordinates of the target area. 5. The target tracking system based on trinocular vision as claimed in claim 1, wherein the target position prediction module is used to predict the coordinates of the target area using the Kalman prediction algorithm to obtain the predicted coordinates of the target area.

Images