CN109190508B - Multi-camera data fusion method based on space coordinate system - Google Patents

Abstract

The invention discloses a multi-camera data fusion method based on a space coordinate system, which comprises the following steps: constructing a training data set aiming at a target to be extracted, and finishing the training of a target detection and recognition model; extracting the category of a target in video data acquired by each camera and the position information of the target under a two-dimensional image coordinate system, and establishing a coordinate mapping relation between the two-dimensional image coordinate system and a three-dimensional space coordinate system; carrying out target detection and target identification processing on video stream data acquired by cameras in a plurality of continuous scenes, and extracting the category information of targets appearing in each frame and the position information of the targets in a two-dimensional image coordinate system; mapping the position information of the target in a two-dimensional image coordinate system into the coordinate of the target in a three-dimensional space coordinate system; and obtaining the motion trail data of the target under the three-dimensional space coordinate system according to the distance information between each target in the current time node and each target in the previous time node.

Description

A multi-camera data fusion method based on spatial coordinate system

technical field

The invention relates to the field of video image processing, in particular to a monitoring data fusion method collected by multiple cameras in a continuous multi-monitoring scene under a spatial coordinate system.

Background technique

In recent years, with the popularization of network cameras for security, intelligent video surveillance technology has rapidly become a research hotspot. Video data is a record of what happened in the surveillance scene, which contains various kinds of information. In view of the fact that the background is fixed in most video surveillance data, what users are really interested in is the target appearing in the video surveillance data and the running track of the target.

At present, most of the video surveillance data are stored separately according to the number of the cameras, and the data collected by each camera is then analyzed and processed by the intelligent video surveillance technology. The target detection, target recognition and target tracking are three important links in the analysis and processing of intelligent video surveillance. Among them, target tracking is used to determine the continuous position of the target we are interested in in the video sequence, and target tracking technology is a basic technology in the field of computer vision, which has a wide range of application values.

The traditional target tracking technology is based on the two-dimensional image space to record the historical trajectory of the target. This method is easy to implement, and can record the trajectory of the target in the current scene. However, the shortcomings of this method are: 1. The traditional target tracking technology is limited to the two-dimensional image space and cannot reflect the position change information of the target in the space; 2. When the target moves across scenes, the traditional target tracking technology needs to The target in the current scene is compared with the targets in the subsequent scenes, and the moving direction of the target is judged based on this, which cannot be used to continuously track the target in multiple scenes.

SUMMARY OF THE INVENTION

The purpose of the present invention is: by establishing the mapping relationship between the two-dimensional image coordinate system and the real three-dimensional space coordinate system, the target obtained after target detection and target recognition will be performed on the video frames obtained by multiple cameras in the two-dimensional image. The coordinate information in the coordinate system is mapped into the space coordinate system, and on this basis, a cross-scene tracking method for the target is provided to realize the fusion of video surveillance data collected by multiple cameras.

On the basis of analyzing the camera imaging principle and camera calibration technology, the method establishes the coordinate mapping equation between the two-dimensional image coordinate system and the real three-dimensional space coordinate system; The coordinates in the image coordinate system are converted into the coordinates of the target in the three-dimensional space coordinate system; finally, the target tracking is performed based on the coordinates of the target in the three-dimensional space coordinate system to realize multi-camera data fusion. This method can restore the position change information of the target in the real space, and can also better track the target across scenes, realize the fusion of data collected by multiple cameras, and provide more high-level target behavior analysis. Information.

In order to achieve the above purpose, the present invention provides a multi-camera data fusion method based on a spatial coordinate system, the method comprising the following steps:

Deploy each camera in the scene to be monitored; build a training data set for the target to be extracted, and complete the training of the target detection and recognition model; The position information of the target in the two-dimensional image coordinate system, and the coordinate mapping relationship between the two-dimensional image coordinate system and the three-dimensional space coordinate system is established; for the video stream data collected by cameras in multiple consecutive scenes, the target detection and target Recognition processing, extracting the category information of the target appearing in each frame and the position information of the target in the two-dimensional image coordinate system; based on the coordinate mapping relationship, the position information of the target in the two-dimensional image coordinate system is mapped to the target in the three-dimensional space coordinate. and according to the distance information between each target in the current time node and each target in the previous time node, find the same target in adjacent time nodes, and connect them to get the coordinates of the target in three-dimensional space The motion trajectory data under the system.

Preferably, the step of establishing the coordinate mapping relationship between the two-dimensional image coordinate system and the three-dimensional space coordinate system specifically includes:

For the scene monitored by each camera, select 50 uniformly distributed pixel coordinates in the two-dimensional image coordinate system, denoted as {(u ₁ ,v ₁ ),(u ₂ ,v ₂ ),...,(u ₅₀ , v ₅₀ )}, and obtain the corresponding spatial coordinates of these 50 pixel coordinates in the real three-dimensional spatial coordinate system, denoted as {(X ₁ , Y ₁ , Z ₁ ), (X ₂ , Y ₂ , Z ₂ ),… ..., (X ₅₀ , Y ₅₀ , Z ₅₀ )}, the coordinates of each point in the real three-dimensional space coordinate system can be obtained by manual measurement or GPS positioning;

In the process of camera imaging, the two-dimensional image coordinate point (u, v) and the corresponding three-dimensional space coordinate point (X, Y, Z) satisfy:

Considering that the targets we are concerned about are all on the ground plane, that is, Z=0, the above formula can be transformed into:

和

最小的参数 (w₁,w₂,b₁,w₃,w₄,b₂)，即求解方程组：Among them, the parameters (w ₁ ,w ₂ ,b ₁ ,w ₃ ,w ₄ ,b ₂ ) are constants, and through the least squares method, find the sum of squares of deviations between all fitting results and the actual data

and

The smallest parameters (w ₁ ,w ₂ ,b ₁ ,w ₃ ,w ₄ ,b ₂ ), that is, solve the system of equations:

Preferably, according to the distance information between each target in the current time node and each target in the previous time node, find the same target in adjacent time nodes, and connect them to obtain the target in the three-dimensional space coordinate system. Motion trajectory data steps, including:

Read the coordinate data of the target in the three-dimensional space coordinate system in the monitoring scene of each camera, and sort the coordinate data according to the time of occurrence of the coordinate data;

Calculate the coordinates of each target in the (i+1)th time node in the three-dimensional space coordinate system ((i+1) ₁ ~ (i+1) _a ) and each target in the i-th time node in the three-dimensional space coordinate system The distance between the coordinates (i ₁ ~ i _b ) under , denoted as d _xy (where x∈[1,a], y∈[1,b]);

When x is a fixed value, temporarily consider that y and x that make d _xy the smallest are the numbers of the same target in two adjacent time nodes;

If d _xy is less than the given threshold T, then determine that x and y are the numbers of the same target in two adjacent time nodes, otherwise x is the first time node that appears in the monitoring scene at the (i+1)th time node. the number of the new target;

Connect the coordinates of the same target in the two adjacent time nodes in the three-dimensional space coordinate system in turn, and record the target category and start and end time nodes to which each coordinate connection line belongs, and obtain the three-dimensional target in the scene monitored by each camera. Motion trajectory data in the space coordinate system.

Preferably, the training steps of the target detection and recognition model are completed, which specifically includes:

Step 201: For each target that needs to be extracted, collect 500-1000 pictures containing the target, and these pictures should include targets shot from different angles as much as possible;

Step 202: Add a category label to the picture collected in step 201 by manually labeling;

Step 203: Add the position information of the target in the two-dimensional image coordinate system to the picture collected in step 201 by manually marking;

Step 204: Randomly scramble the image data set used for training to construct a training data set, and import it into the Caffe-based deep learning framework Faster-RCNN;

Step 205: Modify the training parameters according to the category and the number of categories of the training set, and set the number of iterations;

Step 206: Train a target detection and recognition model.

Preferably, based on the target detection and recognition model, the steps of extracting the category of the target in the video data collected by each camera and the position information of the target under the two-dimensional image coordinate system specifically include:

For the video stream data collected by cameras in multiple consecutive scenes, the target detection and recognition model is used for target detection and target recognition processing, and the category information of the target in the current frame collected by each camera is extracted every 1 second. The position information in the two-dimensional image coordinate system; at the same time, the ID of the camera to which the target belongs and the acquisition time are obtained, and the position information of the target constitutes the target information.

In view of the actual needs of the monitoring scene, the present invention uses the deep learning framework to train the target detection and target recognition model, and can simultaneously extract the category information of multiple different targets and the target in the two images from the images collected by the cameras in the continuous monitoring scenes. The position information under the 2D image coordinate system; through the least squares method, the coordinate mapping relationship between the 2D image coordinate system and the 3D space coordinate system can be established, and the position information of each detected target in the 2D image coordinate system can be obtained. Convert it to the coordinates in the three-dimensional space coordinate system of the target; obtain the motion trajectory of the target in the three-dimensional space coordinate system by calculating the distance information between the targets under the adjacent time nodes. Compared with the traditional target tracking technology, this method can fuse the scattered monitoring data collected by each camera, and can simultaneously obtain the spatial position change information of multiple targets in multiple consecutive scenes; through the coordinate mapping equation, the two-dimensional transformation can be completed. The conversion of pixel coordinates to three-dimensional space coordinates realizes the tracking of the target in three-dimensional space; at the same time, the category data and trajectory data of the target stored in this method are much larger than the continuous image data stored in the traditional target tracking technology. The amount of stored data is reduced and the computational efficiency is improved.

Description of drawings

1 is a schematic flowchart of a multi-camera data fusion method based on a spatial coordinate system provided by an embodiment of the present invention;

Figure 2 is a flow chart of the training process of the target detection and recognition model.

Detailed ways

A multi-camera data fusion method based on a spatial coordinate system of the present invention is further described in detail with reference to the accompanying drawings:

FIG. 1 is a schematic flowchart of a multi-camera data fusion method based on a spatial coordinate system provided by an embodiment of the present invention. As shown in Figure 1, the multi-camera data fusion method based on the spatial coordinate system includes steps S1-S6:

S1. Deploy multiple cameras in the scene to be monitored, wherein the edges of the scene covered by adjacent cameras should be as close as possible, and the edges of the scene covered by adjacent cameras should overlap as little as possible.

S2, training the target detection and recognition model, the model training flow chart is shown in Figure 2, and the specific implementation steps are as follows:

Step 201: For each category, select 500-1000 pictures that contain at least one target of this category. For the selection of pictures, pictures taken at different angles and pictures containing targets with different poses should be selected as much as possible to form a picture data set.

Step 202 : adding a category label to the picture collected in step 201 by manual labeling, where the category label is the category to which the target in the picture belongs.

Step 203: Add the position information of the target in the two-dimensional image coordinate system to the picture collected in step 201 by the method of manual marking, and the position information of the target in the two-dimensional image coordinate system is the coordinates of the rectangular bounding box where the target is located. Information (x1, y1, x2, y2), where (x1, y1) is the upper left corner coordinate of the rectangular bounding box where the target is located, and (x2, y2) is the lower right corner coordinate of the rectangular bounding box where the target is located.

Step 204: Randomly scramble the image data set used for training to construct a training data set, divide it into a training set, a test set and a verification set according to the ratio of 7:2:1, and import the deep learning framework Faster-RCNN.

Step 205: Modify the training parameters according to the target category in the training set and the number of target categories, and set the number of iterations, including the first stage of rpn, the first stage of fast rcnn, the second stage of rpn, and the second stage of fast rcnn. number of iterations.

Step 206 : train the target detection and recognition model, after completing the set number of iterations, a target detection and recognition model file with a suffix of .caffemodel will be obtained.

S3. For the video stream data collected by cameras in multiple consecutive scenes, perform target detection and target recognition processing on the current frames collected by each camera every 1 second. Extract the category information of the target in the current frame and the two-dimensional pixel coordinate position information (u _i , v _i ) of the target in the two-dimensional image coordinate system, and also need to store the ID of the camera to which the target belongs and the time information of the target appearance. These information Information that together constitute a goal.

S4. The specific implementation steps of the coordinate mapping equation calculation method between the two-dimensional coordinates and the three-dimensional coordinates are as follows: in the process of camera imaging, the two-dimensional image coordinate points (u, v) and the corresponding three-dimensional space coordinate points (X, Y, Z) satisfy the following relationship:

Considering that the targets we focus on are all on the ground plane, that is, Z=0, formula (1) can be transformed into:

基于此，可以通过最小二乘法建立二维图像坐标系与三维空间坐标系之间的坐标映射关系，具体步骤如下：in,

Based on this, the coordinate mapping relationship between the two-dimensional image coordinate system and the three-dimensional space coordinate system can be established by the least squares method. The specific steps are as follows:

Step 401 : For the scene monitored by each camera, select 50 pixel coordinates in a uniformly distributed two-dimensional image coordinate system, denoted as {(u ₁ ,v ₁ ),(u ₂ ,v ₂ ),...,(u ₅₀ , v ₅₀ )}, and at the same time collect the corresponding spatial coordinates of these 50 pixel coordinates in the real three-dimensional space coordinate system by manual measurement or GPS positioning, denoted as {(X ₁ , Y ₁ , Z ₁ ), (X ₂ , Y ₂ , Z ₂ ), ..., (X ₅₀ , Y ₅₀ , Z ₅₀ )}.

Step 402: Select appropriate parameters (w ₁ , w ₂ , b ₁ , w ₃ , w ₄ , b ₂ ) through the least squares method to ensure the square sums of deviations between all fitting results and the actual data M _X , M _Y Minimum, M _X , M _Y can be expressed as:

Find the values of the parameters (w ₁ ,w ₂ ,b ₁ ,w ₃ ,w ₄ ,b ₂ ) at the time of minimum value so that M _X and M _Y can be obtained, that is, to solve the equation system:

Bring the parameters (w ₁ ,w ₂ ,b ₁ ,w ₃ ,w ₄ ,b ₂ ) into formula (2) to obtain the mapping equation between the two-dimensional image coordinate system and the three-dimensional space coordinate system in the scene monitored by the camera .

S5. Input the two-dimensional pixel coordinate position information (u _i , v _i ) of each target extracted in step S3 under the two-dimensional image coordinate system into the difference between the two-dimensional image coordinate system and the three-dimensional space coordinate system calculated in step 4 The mapping equation between:

Thereby, the coordinates (X _i , Y _i , Z _i ) of the detected targets in each monitoring scene in the three-dimensional space coordinate system are obtained.

Step S6, multi-camera data fusion, based on the coordinates of the target in each monitoring scene in the three-dimensional space coordinate system, extract the trajectory data of the target in the three-dimensional space coordinate system, and the specific steps are as follows:

Step 601: Read the coordinates of the target in the three-dimensional space coordinate system in each monitoring scene in two adjacent time nodes in sequence according to the order of appearance time.

Step 602: Calculate the coordinates of each target in the (i+1) th time node in the three-dimensional space coordinate system (denoted as (i+1) ₁ to (i+1) _a ) and the coordinates of each target in the ith time node The distance between the coordinates (denoted as i ₁ to i _b ) in the three-dimensional space coordinate system is denoted as d _xy (where x∈[1,a], y∈[1,b]).

Step 603: When x is a fixed value, select y that minimizes d _xy , and temporarily consider x and y to be the numbers of the same target in two adjacent time nodes.

Step 604: If d _xy is less than the given threshold T, then determine that x and y are the numbers of the same target in two adjacent time nodes, otherwise x is the first time that the (i+1)th time node appears in the monitoring system. The number of the new target in the scene.

Step 605: Connect the coordinates of the same target in the three-dimensional space coordinate system in the two adjacent time nodes determined in step 604 in turn, and add the category and start and end of the target to which the connection belongs to each coordinate connection line. Time nodes, these data together constitute the movement trajectory data of the target in the three-dimensional space coordinate system in the scene monitored by each camera.

The embodiment of the present invention can restore the position change information of the target in the real space, and can also better track the target across scenes, realize the fusion of target information collected by multiple cameras, and provide high-level target behavior analysis. more information.

While particular embodiments of this invention have been shown and described, it will be apparent to those skilled in the art that changes and modifications based on the teachings herein can be made without departing from the exemplary embodiments of this invention and its broader aspects . Therefore, the appended claims are intended to include within their scope all such changes and modifications without departing from the true spirit and scope of the exemplary embodiments of this invention.

Claims (5)

1. a multi-camera data fusion method based on a space coordinate system, is characterized in that, comprises the following steps: Deploy each camera in the scene that needs to be monitored; A training data set is constructed for the target to be extracted to complete the training of the target detection and recognition model; based on the target detection and recognition model, the category of the target in the video data collected by each camera and the position information of the target in the two-dimensional image coordinate system are extracted. , to establish the coordinate mapping relationship between the two-dimensional image coordinate system and the three-dimensional space coordinate system; For the video stream data collected by cameras in multiple consecutive scenes, perform target detection and target recognition processing, and extract the category information of the target appearing in each frame and the position information of the target in the two-dimensional image coordinate system; Based on the coordinate mapping relationship, the position information of the target in the two-dimensional image coordinate system is mapped to the coordinates of the target in the three-dimensional space coordinate system; and according to the distance between each target in the current time node and each target in the previous time node information, find the same target in adjacent time nodes, and connect them to obtain the movement trajectory data of the target in the three-dimensional space coordinate system. 2. The method according to claim 1, wherein the step of establishing the coordinate mapping relationship between the two-dimensional image coordinate system and the three-dimensional space coordinate system specifically comprises: For the scene monitored by each camera, select 50 uniformly distributed pixel coordinates in the two-dimensional image coordinate system, denoted as {(u ₁ ,v ₁ ),(u ₂ ,v ₂ ),...,(u ₅₀ , v ₅₀ )}, and obtain the corresponding spatial coordinates of these 50 pixel coordinates in the real three-dimensional spatial coordinate system, denoted as {(X ₁ , Y ₁ , Z ₁ ), (X ₂ , Y ₂ , Z ₂ ),… ..., (X ₅₀ , Y ₅₀ , Z ₅₀ )}, the coordinates of each point in the real three-dimensional space coordinate system can be obtained by manual measurement or GPS positioning; In the process of camera imaging, the two-dimensional image coordinate point (u, v) and the corresponding three-dimensional space coordinate point (X, Y, Z) satisfy:

Considering that the targets we are concerned about are all on the ground plane, that is, Z=0, the above formula can be transformed into:

Among them, the parameters (w ₁ ,w ₂ ,b ₁ ,w ₃ ,w ₄ ,b ₂ ) are constants, and through the least squares method, find the sum of squares of deviations between all fitting results and the actual data

and

The smallest parameters (w ₁ ,w ₂ ,b ₁ ,w ₃ ,w ₄ ,b ₂ ), that is, solve the system of equations:

3. The method according to claim 1, wherein, according to the distance information between each target in the current time node and each target in the previous time node, the same target in the adjacent time node is found, and Connect them to obtain the movement trajectory data steps of the target in the three-dimensional space coordinate system, including: Read the coordinate data of the target in the three-dimensional space coordinate system in the monitoring scene of each camera, and sort the coordinate data according to the time of occurrence of the coordinate data; Calculate the coordinates of each target in the (i+1)th time node in the three-dimensional space coordinate system ((i+1) ₁ ~ (i+1) _a ) and each target in the i-th time node in the three-dimensional space coordinate system The distance between the coordinates (i ₁ ~ i _b ) below is denoted as d _xy , where x∈[1,a], y∈[1,b]; When x is a fixed value, temporarily consider that y and x that make d _xy the smallest are the numbers of the same target in two adjacent time nodes; If d _xy is less than the given threshold T, then determine that x and y are the numbers of the same target in two adjacent time nodes, otherwise x is the first time node that appears in the monitoring scene at the (i+1)th time node. the number of the new target; Connect the coordinates of the same target in the two adjacent time nodes in the three-dimensional space coordinate system in turn, and record the target category and start and end time nodes to which each coordinate connection line belongs, and obtain the three-dimensional target in the scene monitored by each camera. Motion trajectory data in the space coordinate system. 4. The method according to claim 1, wherein the training data set is constructed for the target that needs to be extracted, and the training steps of the target detection and recognition model are completed, specifically comprising: Step 201: For each target that needs to be extracted, collect 500-1000 pictures containing the target, and these pictures should include targets shot from different angles; Step 202: adding a category label to the picture collected in step 201 by manually labeling; Step 203: Add the position information of the target in the two-dimensional image coordinate system to the picture collected in step 201 by manually marking; Step 204: Randomly scramble the image data set used for training to construct a training data set, and import the Caffe-based deep learning framework Faster-RCNN; Step 205: Modify the training parameters according to the category and the number of categories of the training set, and set the number of iterations; Step 206: Train a target detection and recognition model. 5. method according to claim 1, is characterized in that, described extracting the category of the target in the video data that each camera collects and the position information step of target under two-dimensional image coordinate system based on target detection and recognition model, Specifically include: For the video stream data collected by cameras in multiple consecutive scenes, the target detection and recognition model is used for target detection and target recognition processing, and the category information of the target in the current frame collected by each camera is extracted every 1 second. The position information in the two-dimensional image coordinate system; at the same time, the ID of the camera to which the target belongs and the acquisition time are obtained, and the position information of the target constitutes the target information.

Images