CN112017222B

CN112017222B - Video panorama stitching and three-dimensional fusion method and device

Info

Publication number: CN112017222B
Application number: CN202010933528.XA
Authority: CN
Inventors: 白刚; 彭靖轩; 籍盖辉; 李晓波
Original assignee: Innovisgroup Co ltd
Current assignee: Innovisgroup Co ltd
Priority date: 2020-09-08
Filing date: 2020-09-08
Publication date: 2024-08-02
Anticipated expiration: 2040-09-08
Also published as: CN112017222A

Abstract

The embodiment of the application provides a method and a device for video panorama stitching and three-dimensional fusion, wherein the method comprises the following steps: acquiring real-time video pictures of a plurality of adjacent cameras with overlapping areas, and performing image preprocessing on the real-time video pictures; performing feature extraction and feature matching on the real-time video pictures subjected to the image preprocessing, determining matching feature points of two adjacent real-time video pictures, and determining a transformation relation of the two adjacent real-time video pictures according to the matching feature points; performing color difference optimization processing on the two adjacent real-time video pictures, and deforming the real-time video pictures subjected to the color difference optimization processing according to a transformation relation to obtain corresponding panoramic images; rendering the panoramic image to a corresponding position of a preset three-dimensional model in real time through a three-dimensional fusion technology for display; the application can effectively solve the problem of video discontinuity and increase the video readability.

Description

Video panorama stitching and three-dimensional fusion method and device

Technical Field

The application relates to the field of video processing, in particular to a method and a device for video panorama stitching and three-dimensional fusion.

Background

With the environment of people for life and work is a constant increase in the safety requirements of (a), the importance of security systems for buildings is becoming more and more prominent. The video monitoring system is more and more valued by people because of the characteristics of intuitiveness, convenience and abundant information content, and therefore, the video becomes an important component of the security system.

In view of the above-mentioned drawbacks of conventional monitoring in important bayonets or areas, there is a need to propose an improved way, and in recent years, video monitoring technology has been rapidly developed along with the development of computers, networks, image processing, computer graphics and transmission technologies. Many monitoring at present still shows on the screen with 9 palace check or more of little video according to the mode of single camera, and though camera coverage is wide, the picture of every camera exists incoherently, and the detail is lost easily, monitors many problems such as dead angle is many, geographical position is ambiguous, therefore provides the machine that can take advantage of for criminals.

Disclosure of Invention

Aiming at the problems in the prior art, the application provides the video panorama stitching and three-dimensional fusion method and device, which can effectively solve the problem of video discontinuity, increase the video readability, enable users to master the whole situation in real time, avoid leaking through each monitoring corner and have great practical application value.

In order to solve at least one of the problems, the application provides the following technical scheme:

in a first aspect, the present application provides a method for video panorama stitching and three-dimensional fusion, including:

acquiring real-time video pictures of a plurality of adjacent cameras with overlapping areas, and performing image preprocessing on the real-time video pictures;

Performing feature extraction and feature matching on the real-time video pictures subjected to the image preprocessing, determining matching feature points of two adjacent real-time video pictures, and determining a transformation relation of the two adjacent real-time video pictures according to the matching feature points;

Performing color difference optimization processing on the two adjacent real-time video pictures, and deforming the real-time video pictures subjected to the color difference optimization processing according to a transformation relation to obtain corresponding panoramic images;

And rendering the panoramic image to a corresponding position of a preset three-dimensional model in real time through a three-dimensional fusion technology for display.

Further, the acquiring real-time video pictures of a plurality of adjacent cameras with overlapping areas, and performing image preprocessing on the real-time video pictures, includes:

carrying out average weighting treatment on the values of all pixel points of the real-time video picture according to a preset two-dimensional Gaussian filtering kernel function;

And carrying out graying treatment on the real-time video picture subjected to the average weighting treatment to obtain a corresponding gray image.

Further, the feature extraction and feature matching are performed on the real-time video frames after the image preprocessing, the matching feature points of two adjacent real-time video frames are determined, and the transformation relationship of the two adjacent real-time video frames is determined according to the matching feature points, including:

Extracting the characteristics of the real-time video picture to obtain corresponding characteristic points and characteristic descriptors;

performing rough matching on each characteristic point, and determining the Hamming distance between two characteristic points according to the characteristic descriptors;

performing fine matching according to the Hamming distance to obtain matching feature points of two adjacent real-time video pictures;

and determining and storing the transformation relation of the two adjacent real-time video pictures according to the matching characteristic points and a preset random sampling consistency algorithm.

Further, the performing color difference optimization processing on the two adjacent real-time video frames, and deforming the real-time video frames after the color difference optimization processing according to a transformation relationship to obtain corresponding panoramic images, including:

Performing color correction according to the color correction parameters between two adjacent real-time video pictures and preset global adjustment parameters;

establishing panoramic images according to the number of cameras and the resolution of video pictures;

And overlapping and optimizing the overlapping areas of the two adjacent real-time video pictures in the panoramic image to obtain the panoramic image after overlapping and optimizing treatment.

Further, the rendering the panoramic image to the corresponding position of the preset three-dimensional model for display through the three-dimensional fusion technology in real time includes:

determining a three-dimensional model of a target area and a plurality of discrete point pairs of the panoramic image, wherein the discrete point pairs are composed of one three-dimensional model point coordinate and one panoramic image gridding coordinate;

and determining the mapping relation of the panoramic image according to the discrete point pairs, carrying out coordinate interpolation according to the mapping relation, and carrying out panoramic video sampling according to the coordinate interpolation to obtain a three-dimensional fusion image of the panoramic video.

In a second aspect, the present application provides a device for splicing and three-dimensional fusion of video panorama, comprising:

the image preprocessing module is used for acquiring real-time video pictures of a plurality of adjacent cameras with overlapping areas and carrying out image preprocessing on the real-time video pictures;

The transformation relation determining module is used for carrying out feature extraction and feature matching on the real-time video pictures subjected to the image preprocessing, determining matching feature points of two adjacent real-time video pictures, and determining transformation relation of the two adjacent real-time video pictures according to the matching feature points;

the panoramic image generation module is used for carrying out color difference optimization processing on the two adjacent real-time video pictures and deforming the real-time video pictures subjected to the color difference optimization processing according to a transformation relation to obtain corresponding panoramic images;

And the three-dimensional fusion module is used for rendering the panoramic image to the corresponding position of the preset three-dimensional model in real time through a three-dimensional fusion technology for display.

Further, the image preprocessing module includes:

The image noise reduction unit is used for carrying out average weighting processing on the values of all pixel points of the real-time video picture according to a preset two-dimensional Gaussian filter kernel function;

And the image graying unit is used for graying the real-time video picture subjected to the average weighting treatment to obtain a corresponding gray image.

Further, the transformation relation determining module includes:

The feature extraction unit is used for extracting features of the real-time video picture to obtain corresponding feature points and feature descriptors;

the rough matching unit is used for carrying out rough matching on the characteristic points and determining the hamming distance between the two characteristic points according to the characteristic descriptors;

The fine matching unit is used for carrying out fine matching according to the Hamming distance to obtain matching characteristic points of two adjacent real-time video pictures;

and the transformation relation calculating unit is used for determining and storing the transformation relation of the two adjacent real-time video pictures according to the matching characteristic points and a preset random sampling coincidence algorithm.

Further, the panoramic image generation module includes:

the color correction unit is used for carrying out color correction according to the color correction parameters between two adjacent real-time video pictures and the preset global adjustment parameters;

The panoramic image establishing unit is used for establishing panoramic images according to the number of the cameras and the resolution of the video pictures;

And the overlapping optimization unit is used for carrying out overlapping optimization on the overlapping areas of the two adjacent real-time video pictures in the panoramic image to obtain the panoramic image after the overlapping optimization processing.

Further, the three-dimensional fusion module includes:

a discrete point pair determining unit configured to determine a three-dimensional model of a target area and a plurality of discrete point pairs of the panoramic image, wherein the discrete point pairs are composed of one three-dimensional model point coordinate and one panoramic image rasterized coordinate;

and the three-dimensional fusion unit is used for determining the mapping relation of the panoramic image according to the discrete point pairs, carrying out coordinate interpolation according to the mapping relation, and carrying out panoramic video sampling according to the coordinate interpolation to obtain a three-dimensional fusion image of the panoramic video.

In a third aspect, the present application provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor, where the processor implements the steps of the video panorama stitching and three-dimensional fusion method when the program is executed by the processor.

In a fourth aspect, the present application provides a computer readable storage medium having stored thereon a computer program which, when executed by a processor, performs the steps of the video panorama stitching and three-dimensional fusion method.

According to the technical scheme, the video panorama splicing and three-dimensional fusion method and device are provided, a plurality of adjacent camera pictures with overlapping areas are spliced into a complete picture through a panorama splicing technology by means of computer vision and image processing technology, then the real-time spliced picture is rendered to a corresponding three-dimensional model position by means of computer graphics, and the combination of geographic positions and real-time panoramic videos is realized, so that related security personnel can control the monitoring situation of the whole scene at any time and any place, and the riding of criminals is reduced.

Drawings

In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings that are required in the embodiments or the description of the prior art will be briefly described, and it is obvious that the drawings in the following description are some embodiments of the present application, and other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.

FIG. 1 is a flow chart of a video panorama stitching and three-dimensional fusion method according to an embodiment of the present application;

FIG. 2 is a second flow chart of a video panorama stitching and three-dimensional fusion method according to an embodiment of the present application;

FIG. 3 is a third flow chart of a video panorama stitching and three-dimensional fusion method according to an embodiment of the present application;

FIG. 4 is a flowchart of a method for video panorama stitching and three-dimensional fusion according to an embodiment of the present application;

FIG. 5 is a flowchart of a video panorama stitching and three-dimensional fusion method according to an embodiment of the present application;

FIG. 6 is a block diagram of a video panorama stitching and three-dimensional fusion apparatus according to an embodiment of the present application;

FIG. 7 is a second block diagram of a video panorama stitching and three-dimensional fusion apparatus according to an embodiment of the present application;

FIG. 8 is a third block diagram of a video panorama stitching and three-dimensional fusion apparatus according to an embodiment of the present application;

FIG. 9 is a diagram illustrating a structure of a video panorama stitching and three-dimensional fusion device according to an embodiment of the present application;

FIG. 10 is a fifth block diagram of a video panorama stitching and three-dimensional fusion apparatus according to an embodiment of the present application;

FIG. 11 is a schematic diagram of parameters of a Gaussian filtered 3x3 template in accordance with an embodiment of the application;

FIG. 12 is a schematic view of feature matching of a video frame adjacent to a scene in an embodiment of the application;

FIG. 13 is a schematic view of two-dimensional images of panorama stitching of adjacent video frames of a scene in an embodiment of the present application;

FIG. 14 is a schematic diagram showing the effects of real-time video panorama stitching and three-dimensional fusion according to an embodiment of the present application;

FIG. 15 is a second schematic view of the effects of real-time video panorama stitching and three-dimensional fusion according to an embodiment of the present application;

fig. 16 is a schematic structural diagram of an electronic device according to an embodiment of the present application.

Detailed Description

For the purpose of making the objects, technical solutions and advantages of the embodiments of the present application more apparent, the technical solutions of the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application, and it is apparent that the described embodiments are some embodiments of the present application, but not all embodiments of the present application. All other embodiments, which can be made by those skilled in the art based on the embodiments of the application without making any inventive effort, are intended to be within the scope of the application.

In view of the recent development of computers, networks, image processing, computer graphics, and transmission technologies, video surveillance technology has also been rapidly developed. The application provides a video panorama splicing and three-dimensional fusion method and device, which are used for splicing a plurality of adjacent camera pictures with overlapping areas into a complete picture through panorama splicing technology by computer vision and image processing technology, and then rendering the real-time spliced picture to a corresponding three-dimensional model position by computer graphics to realize the combination of geographic position and real-time panoramic video, thereby leading related security personnel to master the monitoring situation of the whole scene at any time and any place and reducing the riding of criminals.

In order to effectively solve the problem of video discontinuity, increase the video readability, make users grasp the overall situation in real time, do not leak through each monitoring corner, have very great practical application value, the application provides an embodiment of a video panorama stitching and three-dimensional fusion method, see fig. 1, the video panorama stitching and three-dimensional fusion method specifically includes the following contents:

Step S101: acquiring real-time video pictures of a plurality of adjacent cameras with overlapping areas, and performing image preprocessing on the real-time video pictures;

Step S102: performing feature extraction and feature matching on the real-time video pictures subjected to the image preprocessing, determining matching feature points of two adjacent real-time video pictures, and determining a transformation relation of the two adjacent real-time video pictures according to the matching feature points;

Step S103: performing color difference optimization processing on the two adjacent real-time video pictures, and deforming the real-time video pictures subjected to the color difference optimization processing according to a transformation relation to obtain corresponding panoramic images;

step S104: and rendering the panoramic image to a corresponding position of a preset three-dimensional model in real time through a three-dimensional fusion technology for display.

As can be seen from the above description, the video panorama stitching and three-dimensional fusion method provided by the embodiment of the present application can stitch a plurality of adjacent camera pictures with overlapping areas into a complete picture through a panorama stitching technology by using computer vision and image processing technologies, and then render the real-time stitched picture to a corresponding three-dimensional model position by using computer graphics, so as to realize the combination of geographic position and real-time panorama video, thereby enabling related security personnel to control the monitoring situation of the whole scene at any time and any place, and reducing the riding of criminals.

In an embodiment of the video panorama stitching and three-dimensional fusion method of the present application, referring to fig. 2, the following may be further specifically included:

step S201: carrying out average weighting treatment on the values of all pixel points of the real-time video picture according to a preset two-dimensional Gaussian filtering kernel function;

step S202: and carrying out graying treatment on the real-time video picture subjected to the average weighting treatment to obtain a corresponding gray image.

Optionally, due to the influence of factors such as production, installation and surrounding environment of the front-end video acquisition equipment, the quality of the acquired video image is different, and the quality of the image directly influences the effect of subsequent image feature extraction and feature matching, so that the image preprocessing is necessary and indispensable in advance, and specifically comprises the following steps: and (5) reducing noise of the image and graying the image.

The method comprises the following steps: since most things in nature are distributed approximately close to gaussian distribution in a sufficient number of samples, and many thousands of pixels are in the image we acquire and each pixel is independent of the other, we use gaussian filtering ground method in the image noise reduction stage. The kernel of a particular gaussian filter can be divided into one dimension, two dimensions up to multiple dimensions, below which we simply introduce the kernel of one-and two-dimensional gaussian filters,

The density function of the one-dimensional gaussian filter is as follows:

wherein μ is the mean and σ is the standard deviation

The kernel function of the two-dimensional gaussian filter is as follows:

Since the image is two-dimensional, we use a two-dimensional gaussian kernel in the image denoising process, and we choose the variance to be 0 in image processing in general. In popular terms, gaussian filtering is a process of carrying out average weighting on the whole image, and the value of each pixel point is obtained by carrying out weighted average on the pixel point and the values of other pixels in the field. The specific operation process is as follows: given each pixel in a scanned image line by line and column by column of a template (otherwise known as a convolution, mask), the value of the center pixel point of the template is replaced with a weighted average of pixels within the field determined by the template. In the present invention we use the convolution template to be a square 3x3 convolution. A specific template design is shown in fig. 11;

Since a gray scale is required in the feature extraction stage, we perform graying of the color image in the second stage of image preprocessing. In this step, we directly use OpenCV self-contained function cvtColor to make graying, so as to obtain a gray image of each video picture.

In an embodiment of the video panorama stitching and three-dimensional fusion method of the present application, referring to fig. 3, the following may be further specifically included:

Step S301: extracting the characteristics of the real-time video picture to obtain corresponding characteristic points and characteristic descriptors;

Step S302: performing rough matching on each characteristic point, and determining the Hamming distance between two characteristic points according to the characteristic descriptors;

Step S303: performing fine matching according to the Hamming distance to obtain matching feature points of two adjacent real-time video pictures;

Step S304: and determining and storing the transformation relation of the two adjacent real-time video pictures according to the matching characteristic points and a preset random sampling consistency algorithm.

Optionally, extracting feature points of the preprocessed images by using a method combining SIFT (Scale-INVARIANT FEATURE TRANSFORM) and SURF (Speeded Up Robust Features) feature extraction to obtain feature points and corresponding feature descriptors of each image;

In the feature matching stage, two matching modes are set for feature matching, namely manual matching and automatic matching, due to the complexity of a scene and the influence of the outside. For automatic matching, performing first matching on the feature points of each image to obtain coarse matching points, and calculating the Hamming distance of each two feature points according to descriptors by using a violent matching algorithm in the matching; then a second exact match is performed, in the present invention the exact match distance threshold used is set to be selectable, the selection range being between 0.4 and 0.8. By the second exact match we obtain the exact match points of the adjacent images. For manual matching, the corresponding characteristic points of the corresponding adjacent images are manually selected directly by using a mouse to perform matching.

And calculating the transformation relation between the adjacent images by utilizing RANSAC (Random Sample Consensus) algorithm according to the calculated matching points of the adjacent images, and storing the transformation relation matrix into a file, so that the next use is convenient.

In an embodiment of the video panorama stitching and three-dimensional fusion method of the present application, referring to fig. 4, the following may be further specifically included:

Step S401: performing color correction according to the color correction parameters between two adjacent real-time video pictures and preset global adjustment parameters;

Step S402: establishing panoramic images according to the number of cameras and the resolution of video pictures;

step S403: and overlapping and optimizing the overlapping areas of the two adjacent real-time video pictures in the panoramic image to obtain the panoramic image after overlapping and optimizing treatment.

Optionally, because the collected images can have chromatic aberration due to factors such as orientation, self-photosensitive components and the like of the adjacent cameras, if the images are directly spliced, macroscopic splicing seams can be generated, and the splicing effect and the use feeling of a user are seriously affected, the invention provides a self-adaptive adjustment method of the chromatic aberration images. Firstly, calculating color correction parameters between adjacent video pictures, wherein the specific practice is as follows: assuming that n images P ₁,P₂,…P_n to be spliced are provided, wherein P _i and P _i+1 are two adjacent video pictures with overlapping areasAnd (3) withIs the overlapping area of the adjacent video frames, the correction parameter of one color channel of the video to-be-spliced image P _i is calculated by the following formula:

in formula (3): m: overlapping regions of adjacent video pictures;

S _i (S): pixel values of pixel points s in the ith image;

S _i+1 (S): pixel values of pixel points s in the (i+1) th image;

Gamma: a specified parameter, typically set to 2.2;

Meanwhile, in order to avoid the image from being over saturated, a global adjusting parameter g _c is set in the invention and is used for adjusting the color value of the whole splicing sequence. Since our video image is generally R, G, B three-channel, it is necessary to calculate the color compensation value and the color adjustment value for each channel, and the specific calculation formula of the color adjustment value is as follows:

finally, the formula for correcting the color of the video picture by the color correction parameters and the global adjustment parameters is as follows:

S_c,i(s)＝(g_cα_c,i)^1/γS_c,i(s) (5)

Wherein S _c,i (S) refers to the pixel value of the pixel point of the video frame P _i on the channel c e { R, G, B }.

S32, building panoramic mapping images according to the number of panoramic stitching cameras, wherein the panoramic images are determined according to the number of cameras and the resolution of video pictures of each camera. Assuming that there are N cameras for panoramic stitching, the video picture resolution for each individual camera is: wxH, then the resolution of the generated panoramic image is: (WxN) xH. If the number of the selected spliced cameras is an odd number, selecting a video picture made by an intermediate camera as a reference picture, wherein the mapping relation between the intermediate camera and the panoramic image is an identity matrix, the x direction (namely the width of the corresponding video) in the translation matrix is the size of the resolution of the video picture, and the y direction (namely the height of the corresponding video) in the translation matrix is 0; the process of generating a panorama is specifically described in one case: assuming that n (n is an odd number) cameras exist, P ₁,P₂,P₃…P_n is respectively used, the transformation matrix between the video pictures of the adjacent cameras obtained by the calculation is as follows: h ₁,H₂,H₃…H_n-1, the subscript of the transformation matrix represents the transformation relationship of the current camera i and the camera i+1 to the right of the current camera. The intermediate camera numbers are found by the above description: (1+n)/2. Let P _(1+n)/2 be the reference camera, then the transformation from the reference camera to the panoramic image is the identity matrix, let I be the translation matrix be T, then the transformation formula from the reference camera pixel point to the panoramic image is as follows:

P_p＝T*I*P_R (6)

Wherein P _p represents the position of a pixel in the panorama and is a homogeneous coordinate;

P _c represents the position of a certain pixel in the reference image, which is a homogeneous coordinate;

I, T represents rotation and translation matrix respectively, is 3x3 matrix;

further, the image to the left of the reference image, namely the sequence number: the transformation relation from each image pixel point to the panorama is calculated by the following formula:

In particular if Then equation (7) becomes:

for the image to the right of the reference image, that is: the conversion relation calculation formula from the pixel point of each image to the panoramic image is as follows:

Wherein the method comprises the steps of

In particular ifThen equation (9) is modified to:

finally, each image can be mapped to the panorama by the above formulas (6) to (10).

And searching a superposition area of the adjacent images in the panoramic image, optimizing the superposition area of the adjacent images in the panoramic image, and generating the panoramic image. If the overlapping area of the adjacent images in the panoramic image is not optimized, the overlapping area in the panoramic image loses some details because the pixels are too saturated, and the pixel calculation formula of the overlapping area of the adjacent video pictures in the panoramic image is as follows:

P_(m,n+0)＝P_i,(m,n+0)*α+P_i+1,(m,n+0)*(1-α) (11)

P_(m,n+1)＝P_i,(m,n+1)*α+P_i+1,(m,n+1)*(1-α) (12)

P_(m,n+2)＝P_i,(m,n+2)*α+P_i+1,(m,n+2)*(1-α) (13)

α＝((width*ratio)-(n-s))/(width*ratio) (14)

wherein P _(m,n+k) represents the pixel value of the k (k=0, 1, 2) th channel overlap region position (m, n) in the panoramic image;

P _i,(m,n+k),P_i+1,(m,n+k) represents the pixel values of the k (k=0, 1, 2) th channel overlap region positions (m, n) in the image i and the image i+1, respectively.

Width represents the width of the overlapping area of adjacent video pictures;

ratio represents the overlapping proportion of overlapping areas of adjacent video pictures;

s represents the initial position of the overlapping area of adjacent video pictures;

After the steps are finished, the panoramic image is generated and then transmitted to the client for configuration display.

In an embodiment of the video panorama stitching and three-dimensional fusion method of the present application, referring to fig. 5, the following may be further specifically included:

step S501: determining a three-dimensional model of a target area and a plurality of discrete point pairs of the panoramic image, wherein the discrete point pairs are composed of one three-dimensional model point coordinate and one panoramic image gridding coordinate;

step S502: and determining the mapping relation of the panoramic image according to the discrete point pairs, carrying out coordinate interpolation according to the mapping relation, and carrying out panoramic video sampling according to the coordinate interpolation to obtain a three-dimensional fusion image of the panoramic video.

Optionally, in order to accurately determine the position of the panoramic video, a static panoramic image is first selected as a reference for editing, and a plurality of binary point pairs are respectively selected from the three-dimensional scene and the corresponding panoramic image, wherein each point pair is composed of a three-dimensional scene point coordinate and a two-dimensional panoramic image rasterized coordinate.

Because the mapping relation of the whole panoramic image needs to be obtained by interpolation according to the discrete point pairs in the steps, selecting a proper interpolation method is important for the final panoramic mapping result, and the specific steps of 2 are as follows:

because the panoramic image is two-dimensional, if interpolation is directly carried out on the corresponding three-dimensional selected points in the three-dimensional space, perspective distortion of interpolation results can be caused, therefore, the invention takes the screen space as the interpolation space, adopts interpolation on the three-dimensional selected points in the screen space, and converts the three-dimensional selected points in the three-dimensional space into two-dimensional points in the screen space through projection transformation.

The method is good for the situation that the distribution of the discrete points is uniform, but the situation that the distribution of the discrete points is uneven usually causes larger distortion of an interpolation image, because the nearest distance judgment can cause the superposition of the adjacent discrete points of a plurality of interpolation points, the interpolation result is overlapped, and the distortion of the interpolation result can be caused when the distribution of the adjacent discrete points is close to a line, therefore, the invention adopts the delaunay triangulation method to divide the space domain formed by all the discrete points, and the delaunay triangulation has good property of maximally generating triangle interior angles, thereby effectively avoiding the generation of an elongated triangle.

The invention adopts triangle interpolation based on barycentric coordinates, takes the triangle generated in the steps as an interpolation domain, determines barycentric coordinates according to the areas of three triangles formed by interpolation points and three vertexes of the triangle, and carries out sampling point interpolation by using the barycentric coordinates.

Although the delaunay triangulation method is stable, there is still possibility that long and narrow triangles exist at the edge of an interpolation area to cause distortion of an interpolation image, so the invention provides an unsupervised long and narrow triangle automatic elimination algorithm, the algorithm firstly judges a group of triangles at the outermost periphery by traversing all triangles of the current triangulation, determines whether the definition of the long and narrow triangles is met according to the ratio of the longest edge of the long and narrow triangles to the height of the edge, deletes the long and narrow triangles if the long and narrow triangles exist, and re-executes the traversing algorithm after the current traversal is finished until the long and narrow triangles are no longer exist in the current traversal, and the algorithm is terminated.

And sampling the panoramic video according to the coordinates obtained by interpolation in the steps to generate final panoramic video three-dimensional fusion.

In order to effectively solve the problem of video discontinuity, increase the video readability, make users master the overall situation in real time, do not leak each monitoring corner, have very great practical application value, the application provides an embodiment of a video panorama stitching and three-dimensional fusion device for implementing all or part of the content of the video panorama stitching and three-dimensional fusion method, see fig. 6, the video panorama stitching and three-dimensional fusion device specifically comprises the following contents:

an image preprocessing module 10, configured to acquire real-time video frames of a plurality of adjacent cameras having overlapping areas, and perform image preprocessing on the real-time video frames;

The transformation relation determining module 20 is configured to perform feature extraction and feature matching on the real-time video frames subjected to the image preprocessing, determine matching feature points of two adjacent real-time video frames, and determine a transformation relation of the two adjacent real-time video frames according to the matching feature points;

The panoramic image generation module 30 is configured to perform color difference optimization processing on the two adjacent real-time video frames, and deform the real-time video frames after the color difference optimization processing according to a transformation relationship to obtain corresponding panoramic images;

the three-dimensional fusion module 40 is configured to render the panoramic image to a corresponding position of a preset three-dimensional model in real time for display through a three-dimensional fusion technology.

As can be seen from the above description, the video panorama stitching and three-dimensional fusion device provided by the embodiment of the present application can stitch a plurality of adjacent camera pictures with overlapping areas into a complete picture through a panorama stitching technology by using computer vision and image processing technologies, and then render the real-time stitched picture to a corresponding three-dimensional model position by using computer graphics, so as to realize the combination of geographic position and real-time panorama video, thereby enabling related security personnel to control the monitoring situation of the whole scene at any time and any place, and reducing the riding machine of criminals.

In an embodiment of the video panorama stitching and three-dimensional fusion device of the present application, referring to fig. 7, the image preprocessing module 10 comprises:

The image noise reduction unit 11 is used for carrying out average weighting processing on the values of all pixel points of the real-time video picture according to a preset two-dimensional Gaussian filtering kernel function;

and the image graying unit 12 is used for graying the real-time video picture subjected to the average weighting treatment to obtain a corresponding gray image.

In an embodiment of the video panorama stitching and three-dimensional fusion device of the present application, referring to fig. 8, the transformation relationship determining module 20 comprises:

a feature extraction unit 21, configured to perform feature extraction on the real-time video frame, so as to obtain corresponding feature points and feature descriptors;

A rough matching unit 22, configured to perform rough matching on each of the feature points, and determine a hamming distance between two of the feature points according to the feature descriptors;

a fine matching unit 23, configured to perform fine matching according to the hamming distance, so as to obtain matching feature points of two adjacent real-time video frames;

and the transformation relation calculating unit 24 is used for determining and storing the transformation relation of the two adjacent real-time video pictures according to the matching characteristic points and a preset random sampling coincidence algorithm.

In an embodiment of the video panorama stitching and three-dimensional fusion device of the present application, referring to fig. 9, the panorama image generation module 30 comprises:

a color correction unit 31, configured to perform color correction according to a color correction parameter between two adjacent real-time video frames and a preset global adjustment parameter;

A panoramic image creation unit 32 for creating a panoramic image according to the number of cameras and the resolution of the video frames;

And the overlap optimizing unit 33 is configured to perform overlap optimization on overlapping areas of the two adjacent real-time video frames in the panoramic image, so as to obtain the panoramic image after the overlap optimization processing.

In an embodiment of the video panorama stitching and three-dimensional fusion apparatus of the present application, referring to fig. 10, the three-dimensional fusion module 40 comprises:

a discrete point pair determining unit 41 for determining a three-dimensional model of a target area and a plurality of discrete point pairs of the panoramic image, wherein the discrete point pairs are composed of one three-dimensional model point coordinate and one panoramic image rasterized coordinate;

the three-dimensional fusion unit 42 is configured to determine a mapping relationship of the panoramic image according to the discrete point pairs, perform coordinate interpolation according to the mapping relationship, and perform panoramic video sampling according to the coordinate interpolation, so as to obtain a three-dimensional fusion image of the panoramic video.

In order to further explain the scheme, the application also provides a specific application example for realizing the video panorama stitching and three-dimensional fusion method by applying the video panorama stitching and three-dimensional fusion device, which comprises the following contents:

A technical method and a process for video panorama stitching and three-dimensional fusion comprise the following steps:

s1, acquiring real-time video pictures of a plurality of adjacent cameras with overlapping areas, and preprocessing images on the real-time video pictures by an image processing technology;

S2, utilizing a feature extraction and matching technology to the real-time video picture to obtain matching feature points of the adjacent video picture, and calculating the transformation relation of the images of the adjacent video picture according to the matching feature points

S3, performing color difference optimization on the adjacent video pictures in the S2, deforming the video pictures according to the transformation relation of the adjacent video pictures, and generating corresponding panoramic pictures;

And S4, loading the three-dimensional model by the client, taking the panoramic image output in the S3 as a texture, and displaying the panoramic image to a corresponding position of the model in real time through the three-dimensional fusion technology.

According to the invention, the plurality of camera video pictures with the overlapping areas are spliced in a seamless manner, and the GPU is utilized to accelerate image processing, so that the output panoramic picture is smooth and free of blocking, and the requirement of watching by human eyes is met. Finally, the panoramic video is rendered into the three-dimensional model in real time by utilizing the GPU technology through a special means, so that perfect fusion and display of the virtual model and the real video are realized.

The features and capabilities of the present invention are described in further detail below in connection with the following examples:

s1, acquiring real-time video pictures of a plurality of adjacent cameras with overlapping areas, and preprocessing images on the real-time video pictures through an image processing technology:

Taking Xinjiang as an example, firstly opening panoramic generation configuration software, acquiring a plurality of adjacent cameras with overlapping areas through a device list to acquire real-time monitoring pictures, then selecting configuration splicing parameters, defaulting in general, clicking a start button, automatically preprocessing images according to our description by a background, and finally displaying matched features in the pictures.

S2, the real-time video picture utilizes a feature extraction and matching technology to obtain matching feature points of the adjacent video picture, and the transformation relation of the images of the adjacent video picture is calculated according to the matching feature points:

in the field of panoramic stitching, robustness of feature point detection and matching influences consistency of video panoramic stitching to a great extent. The scenario tested in this example is Xinjiang. By using the characteristic point method of combining SURF and SIFT and subsequent automatic matching, the matching points of two adjacent images are finally obtained, as shown in FIG. 12. It is evident that the number of correct matches is the majority and provides a powerful basis for the subsequent continued use of the RANSAC algorithm to calculate the transform relationship between adjacent video pictures.

S3, performing color difference optimization on the adjacent video pictures in the S2. Then deforming the video pictures according to the transformation relation of the adjacent video pictures, and generating corresponding panoramic pictures, wherein the generated panoramic pictures are shown in fig. 13;

And S4, loading the three-dimensional model by the client, taking the panorama output in the step S3 as a texture, and displaying the panorama to a corresponding position of the model in real time by the three-dimensional fusion technology, wherein the final display effect is shown in fig. 14 and 15.

As can be seen from the above, the present application can achieve at least the following technical effects:

1. According to the application, the obtained real-time video pictures of the cameras are subjected to image preprocessing and manual and automatic feature matching realized by dividing the situation into fields, so that on one hand, the feature matching accuracy and speed of the adjacent video pictures are improved, and on the other hand, the application of the method can adapt to various complex environments, and the defects of video splicing are effectively overcome.

2. According to the application, the middle camera video picture is selected as a reference, so that the deformation of the video pictures at two ends in the panorama is effectively improved. Meanwhile, the color difference problem of the ground video picture caused by the influence of the camera and the external environment is regulated by adopting a local color separation channel and a global self-adaptive method, the quality of image splicing and the visual feeling of a user are further improved, and finally, the pixel calculation mode of the overlapping area of the adjacent video picture and the overlapping proportion of the adjacent video picture are set, so that the ghost problem of the spliced picture caused by the installation position of the camera can be obviously reduced.

3. The application effectively improves the perspective distortion of interpolation results caused by direct interpolation in a three-dimensional space by interpolating edit points on a screen space, simultaneously adopts Delaunay triangulation to determine interpolation vertexes, effectively avoids interpolation distortion caused by an elongated triangle, and finally provides an unsupervised automatic elimination algorithm for the elongated triangle possibly existing in an interpolation edge area, thereby ensuring that the finally generated interpolation image cannot be obviously distorted under various conditions.

4. The technical method and the process for video panorama stitching and three-dimensional fusion are suitable for various real environments. The application can enable security personnel to efficiently and conveniently position the whole situation of the control monitoring area to a specific position once danger occurs, and the application realizes the combination of video and geographic position, perfectly solves the problems of video dispersion and undefined geographic position when danger positioning is difficult.

In order to effectively solve the problem of video discontinuity in the hardware aspect and increase the video readability, so that a user can grasp the whole situation in real time and does not leak each monitoring corner, and the embodiment of the electronic equipment for realizing all or part of the content in the video panoramic stitching and three-dimensional fusion method is provided, and the electronic equipment specifically comprises the following contents:

A processor (processor), a memory (memory), a communication interface (Communications Interface), and a bus; the processor, the memory and the communication interface complete communication with each other through the bus; the communication interface is used for realizing information transmission between the video panorama splicing and three-dimensional fusion device and related equipment such as a core service system, a user terminal, a related database and the like; the logic controller may be a desktop computer, a tablet computer, a mobile terminal, etc., and the embodiment is not limited thereto. In this embodiment, the logic controller may refer to an embodiment of the video panorama stitching and three-dimensional fusion method and an embodiment of the video panorama stitching and three-dimensional fusion device in the embodiments, and the contents thereof are incorporated herein, and the repetition is omitted.

It is understood that the user terminal may include a smart phone, a tablet electronic device, a network set top box, a portable computer, a desktop computer, a Personal Digital Assistant (PDA), a vehicle-mounted device, a smart wearable device, etc. Wherein, intelligent wearing equipment can include intelligent glasses, intelligent wrist-watch, intelligent bracelet etc..

In practical applications, part of the video panorama stitching and three-dimensional fusion method may be performed on the electronic device side as described above, or all operations may be performed in the client device. Specifically, the selection may be made according to the processing capability of the client device, and restrictions of the use scenario of the user. The application is not limited in this regard. If all operations are performed in the client device, the client device may further include a processor.

The client device may have a communication module (i.e. a communication unit) and may be connected to a remote server in a communication manner, so as to implement data transmission with the server. The server may include a server on the side of the task scheduling center, and in other implementations may include a server of an intermediate platform, such as a server of a third party server platform having a communication link with the task scheduling center server. The server may include a single computer device, a server cluster formed by a plurality of servers, or a server structure of a distributed device.

Fig. 16 is a schematic block diagram of a system configuration of an electronic device 9600 according to an embodiment of the present application. As shown in fig. 16, the electronic device 9600 may include a central processor 9100 and a memory 9140; the memory 9140 is coupled to the central processor 9100. Notably, this fig. 16 is exemplary; other types of structures may also be used in addition to or in place of the structures to implement telecommunications functions or other functions.

In one embodiment, the video panorama stitching and three-dimensional fusion method functions may be integrated into the central processor 9100. The central processor 9100 may be configured to perform the following control:

As can be seen from the above description, the electronic device provided by the embodiment of the present application uses the computer vision and image processing technology to splice a plurality of adjacent camera images with overlapping areas into a complete image through the panorama stitching technology, and then uses computer graphics to render the real-time stitched image to a corresponding three-dimensional model position, so as to combine the geographic position and the real-time panoramic video, thereby enabling the related security personnel to control the monitoring situation of the whole scene at any time and any place, and reducing the ridable machines of criminals.

In another embodiment, the video panorama stitching and three-dimensional fusion device may be configured separately from the central processor 9100, for example, the video panorama stitching and three-dimensional fusion device may be configured as a chip connected to the central processor 9100, and the functions of the video panorama stitching and three-dimensional fusion method are implemented under the control of the central processor.

As shown in fig. 16, the electronic device 9600 may further include: a communication module 9110, an input unit 9120, an audio processor 9130, a display 9160, and a power supply 9170. It is noted that the electronic device 9600 need not include all of the components shown in fig. 16; in addition, the electronic device 9600 may further include components not shown in fig. 16, and reference may be made to the related art.

As shown in fig. 16, the central processor 9100, sometimes also referred to as a controller or operational control, may include a microprocessor or other processor device and/or logic device, which central processor 9100 receives inputs and controls the operation of the various components of the electronic device 9600.

The memory 9140 may be, for example, one or more of a buffer, a flash memory, a hard drive, a removable media, a volatile memory, a non-volatile memory, or other suitable device. The information about failure may be stored, and a program for executing the information may be stored. And the central processor 9100 can execute the program stored in the memory 9140 to realize information storage or processing, and the like.

The input unit 9120 provides input to the central processor 9100. The input unit 9120 is, for example, a key or a touch input device. The power supply 9170 is used to provide power to the electronic device 9600. The display 9160 is used for displaying display objects such as images and characters. The display may be, for example, but not limited to, an LCD display.

The memory 9140 may be a solid state memory such as Read Only Memory (ROM), random Access Memory (RAM), SIM card, etc. But also a memory which holds information even when powered down, can be selectively erased and provided with further data, an example of which is sometimes referred to as EPROM or the like. The memory 9140 may also be some other type of device. The memory 9140 includes a buffer memory 9141 (sometimes referred to as a buffer). The memory 9140 may include an application/function storage portion 9142, the application/function storage portion 9142 storing application programs and function programs or a flow for executing operations of the electronic device 9600 by the central processor 9100.

The memory 9140 may also include a data store 9143, the data store 9143 for storing data, such as contacts, digital data, pictures, sounds, and/or any other data used by an electronic device. The driver storage portion 9144 of the memory 9140 may include various drivers of the electronic device for communication functions and/or for performing other functions of the electronic device (e.g., messaging applications, address book applications, etc.).

The communication module 9110 is a transmitter/receiver 9110 that transmits and receives signals via an antenna 9111. A communication module (transmitter/receiver) 9110 is coupled to the central processor 9100 to provide input signals and receive output signals, as in the case of conventional mobile communication terminals.

Based on different communication technologies, a plurality of communication modules 9110, such as a cellular network module, a bluetooth module, and/or a wireless local area network module, etc., may be provided in the same electronic device. The communication module (transmitter/receiver) 9110 is also coupled to a speaker 9131 and a microphone 9132 via an audio processor 9130 to provide audio output via the speaker 9131 and to receive audio input from the microphone 9132 to implement usual telecommunications functions. The audio processor 9130 can include any suitable buffers, decoders, amplifiers and so forth. In addition, the audio processor 9130 is also coupled to the central processor 9100 so that sound can be recorded locally through the microphone 9132 and sound stored locally can be played through the speaker 9131.

The embodiment of the present application further provides a computer readable storage medium capable of implementing all the steps in the video panorama stitching and three-dimensional fusion method in which the execution subject is a server or a client, where the computer readable storage medium stores a computer program, and when the computer program is executed by a processor, the computer program implements all the steps in the video panorama stitching and three-dimensional fusion method in which the execution subject is a server or a client, for example, the processor implements the following steps when executing the computer program:

As can be seen from the above description, the computer readable storage medium provided by the embodiment of the present application, through the computer vision and the image processing technology, splices a plurality of adjacent camera pictures with overlapping areas into a complete picture through the panorama splicing technology, and then renders the real-time spliced picture to a corresponding three-dimensional model position by using computer graphics, so as to realize the combination of the geographic position and the real-time panoramic video, thereby enabling the related security personnel to control the monitoring situation of the whole scene at any time and any place, and reducing the riding machine of criminals.

It will be apparent to those skilled in the art that embodiments of the present invention may be provided as a method, apparatus, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.

The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (devices), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

The principles and embodiments of the present invention have been described in detail with reference to specific examples, which are provided to facilitate understanding of the method and core ideas of the present invention; meanwhile, as those skilled in the art will have variations in the specific embodiments and application scope in accordance with the ideas of the present invention, the present description should not be construed as limiting the present invention in view of the above.

Claims

1. A method for video panorama stitching and three-dimensional fusion, the method comprising:

Acquiring real-time video pictures of a plurality of adjacent cameras with overlapping areas, and performing image preprocessing on the real-time video pictures, wherein the method comprises the following steps:

carrying out average weighting treatment on the values of all pixel points of the real-time video picture according to a preset two-dimensional Gaussian filtering kernel function, giving each pixel in a scanning image of a template row by row and column by column, and replacing the value of a central pixel point of the template by using a weighted average value of pixels in the field determined by the template, wherein the template adopts convolution of square 3x 3;

Carrying out graying treatment on the real-time video pictures subjected to the average weighting treatment to obtain corresponding gray images, and carrying out graying treatment by using a function cvtColor carried by OpenCV to obtain the gray images of each video picture;

performing feature extraction and feature matching on the real-time video pictures subjected to the image preprocessing, determining matching feature points of two adjacent real-time video pictures, and determining a transformation relationship of the two adjacent real-time video pictures according to the matching feature points, wherein the method comprises the following steps:

Extracting features of the real-time video picture to obtain corresponding feature points and feature descriptors, extracting the feature points of the images by using a method combining SIFT and SURF feature extraction, and obtaining the feature points and the corresponding feature descriptors of each image;

Setting two matching modes for feature matching, namely manual matching and automatic matching;

For automatic matching, performing first matching on the feature points of each obtained image to obtain rough matching points, performing rough matching on each feature point by using a violent matching algorithm in matching, and determining the hamming distance between the two feature points according to the feature descriptors;

Then performing secondary accurate matching, wherein the used accurate matching distance threshold is set as optional, the selection range is between 0.4 and 0.8, and the accurate matching points of the adjacent images are obtained through the secondary accurate matching, and the accurate matching is performed according to the Hamming distance to obtain the matching feature points of the two adjacent real-time video pictures;

Determining and storing the transformation relation of the two adjacent real-time video pictures according to the matching characteristic points and a preset random sampling coincidence algorithm, wherein the storing is to calculate the transformation relation between the adjacent images by using a RANSAC algorithm and store the transformation relation matrix into a file;

For manual matching, the corresponding characteristic points of the corresponding adjacent images are manually selected directly by using a mouse to perform matching;

Performing color difference optimization processing on the two adjacent real-time video frames, and deforming the real-time video frames after the color difference optimization processing according to a transformation relation to obtain corresponding panoramic images, wherein the method comprises the following steps:

There are n images P ₁,P₂,…P_n to be stitched, where P _i and P _i+1 are two adjacent video pictures with overlapping regions, there are And (3) withIs the overlapping area of the adjacent video frames, the correction parameter of one color channel of the image P _i to be spliced is calculated as follows:

wherein M is the overlapping area of adjacent video pictures, S _i (S) is the pixel value of the pixel point S in the ith image, S _i+1 (S) is the pixel value of the pixel point S in the (i+1) th image, and gamma is a specified parameter and is set to 2.2;

setting a global adjusting parameter G _x to adjust the color value of the whole splicing sequence, and calculating a color compensation value and a color adjusting value for each of the three channels R, G and B respectively, wherein the calculating formula of the color adjusting value is as follows:

S_c,i(s)＝(g_cα_c,i)^1/γS_c,i(s)

wherein S _c,i (S) is the pixel value of the pixel point of the video picture Pi on the channel c E { R, G, B };

According to the number of cameras and the video picture resolution, panoramic images are established, N cameras for panoramic stitching are arranged, and the video picture resolution of each independent camera is as follows: wxH, then the resolution of the generated panoramic image is: (WxN) xH, when the number of the selected spliced cameras is an odd number, selecting a video picture of an intermediate camera as a reference picture, wherein the mapping relation between the intermediate camera and the panoramic image is an identity matrix, the x direction in the translation matrix is the resolution of the video picture, and the y direction in the translation matrix is 0; let n cameras be odd, n is P ₁,P₂,P₃…P_n respectively, the transformation matrix between the adjacent camera video pictures calculated is: h ₁,H₂,H₃…H_n-1, the subscript of the transformation matrix indicates the transformation relationship between the current camera i and the camera i+1 to the right of the current camera, and find the middle camera number as follows: (1+n)/2, assuming that P (1+n)/2 is a reference camera, the transformation from the reference camera to the panoramic image is a unitary matrix, and assuming that I is a translation matrix and T is a translation matrix, the transformation formula from the reference camera pixel point to the panoramic image is as follows:

P_p＝T*I*P_R

Wherein Pp represents the position of a certain pixel in the panoramic image, P _R represents the position of a certain pixel in the reference image, P is the homogeneous coordinate, I and T represent rotation and translation matrices respectively, and are 3x3 matrices;

the image to the left of the reference image, namely the sequence number: the transformation relation from each image pixel point to the panorama is calculated by the following formula:

If it is The transformation relation calculation formula from each image pixel point to the panorama becomes:

wherein,

If it isThe transformation relation calculation formula from the pixel point of each image to the panoramic image is deformed as follows:

Finally, each image can be mapped to the panorama by the formula;

Overlapping optimization is carried out on overlapping areas of the two adjacent real-time video pictures in the panoramic image to obtain the panoramic image after the overlapping optimization treatment, overlapping areas of adjacent images in the panoramic image are searched, overlapping areas of the adjacent images in the panoramic image are optimized, the panoramic image is generated, if the overlapping areas of the adjacent images in the panoramic image are not optimized, the overlapping areas in the panoramic image lose some details due to oversaturation of pixels, and a pixel calculation formula of the overlapping areas of the adjacent video pictures in the panoramic image is as follows:

P_(x,y+0)＝P_i,(x,y+0)*a+P_i+1,(x,y+0)*(1-z)

P_(x,y+1)＝P_i,(x,y+1)*a+P_i+1,(x,y+1)*(1-z)

P_(x,y+2)＝P_i,(x,y+2)*a+P_i+1,(x,y+2)*(1-z)

z＝((width*ratio)-(y-s))/(width*ratio)

Wherein P _(x,y+k) represents the pixel value of the kth (k=0, 1, 2) channel overlapping region position (x, y) in the panoramic image, P _i,(x,y+k),P_i+1,(x,y+k) represents the pixel value of the kth (k=0, 1, 2) channel overlapping region position (x, y) in the image i and the image i+1, respectively, width represents the adjacent video picture overlapping region width, ratio represents the adjacent video picture overlapping region overlapping ratio, and s represents the starting position of the adjacent video picture overlapping region;

rendering the panoramic image to a corresponding position of a preset three-dimensional model in real time for display through a three-dimensional fusion technology, wherein the method comprises the following steps:

Selecting a static panoramic picture as a reference for editing, respectively selecting a plurality of binary point pairs in a three-dimensional scene and a corresponding panoramic picture, wherein each point pair consists of a three-dimensional scene point coordinate and a two-dimensional panoramic picture rasterized coordinate, taking a screen space as an interpolation space, interpolating three-dimensional selected points in the screen space, and converting the three-dimensional selected points in the three-dimensional space into two-dimensional points in the screen space through projection transformation;

2. A video panorama stitching and three-dimensional fusion device, comprising:

the image preprocessing module is used for acquiring real-time video pictures of a plurality of adjacent cameras with overlapping areas and performing image preprocessing on the real-time video pictures, and comprises the following steps: an image noise reduction unit and an image graying unit;

the image noise reduction unit is used for carrying out average weighting processing on the values of all pixel points of the real-time video picture according to a preset two-dimensional Gaussian filtering kernel function, giving each pixel in a scanning image of a template row by column, replacing the value of a central pixel point of the template by using the weighted average value of the pixels in the field determined by the template, wherein the template adopts convolution of square 3x 3;

The image graying unit is used for graying the real-time video pictures subjected to the average weighting treatment to obtain corresponding gray images, and graying is carried out by using a function cvtColor carried by OpenCV to obtain gray images of each video picture;

the transformation relation determining module is used for carrying out feature extraction and feature matching on the real-time video pictures subjected to the image preprocessing, determining matching feature points of two adjacent real-time video pictures, and determining the transformation relation of the two adjacent real-time video pictures according to the matching feature points, and comprises the following steps: the device comprises a feature extraction unit, a coarse matching unit, a fine matching unit, a transformation relation calculation unit and a panoramic image generation module;

the feature extraction unit is used for extracting features of the real-time video picture to obtain corresponding feature points and feature descriptors, extracting the feature points of the images by using a method combining SIFT and SURF feature extraction, and obtaining the feature points and the corresponding feature descriptors of each image;

The rough matching unit is used for carrying out first matching on the feature points of each obtained image to obtain rough matching points, carrying out rough matching on each feature point by using a violent matching algorithm in matching, and determining the hamming distance between the two feature points according to the feature descriptors;

The precise matching unit is used for setting a distance threshold of precise matching to be selectable, the selection range is between 0.4 and 0.8, and precise matching points of adjacent images are obtained through the precise matching of the second time, and the precise matching unit is used for performing precise matching according to the Hamming distance to obtain matching characteristic points of two adjacent real-time video pictures;

The transformation relation calculating unit is used for determining and storing the transformation relation of the two adjacent real-time video pictures according to the matching characteristic points and a preset random sampling coincidence algorithm, wherein the storage is to calculate the transformation relation between the adjacent images by using a RANSAC algorithm and store the transformation relation matrix into a file;

The panoramic image generation module is used for carrying out color difference optimization processing on the two adjacent real-time video pictures, and deforming the real-time video pictures subjected to the color difference optimization processing according to a transformation relation to obtain corresponding panoramic images, and comprises the following steps: the system comprises a color correction unit, a panoramic image establishing unit and an overlap optimizing unit;

A color correction unit for performing color correction according to color correction parameters between two adjacent real-time video frames and preset global adjustment parameters, and provided with n images P ₁,P₂,…P_n to be spliced, wherein P _i and P _i+1 are two adjacent video frames with overlapping regions, and provided with And (3) withIs the overlapping area of the adjacent video frames, the correction parameter of one color channel of the image P _i to be spliced is calculated as follows:

S_c,i(s)＝(g_cα_c,i)^1/γS_c,i(s)

The panoramic image establishing unit is used for establishing panoramic images according to the number of cameras and the video picture resolution, N cameras for panoramic stitching are arranged, and the video picture resolution of each independent camera is as follows: wxH, then the resolution of the generated panoramic image is: (WxN) xH, when the number of the selected spliced cameras is an odd number, selecting a video picture of an intermediate camera as a reference picture, wherein the mapping relation between the intermediate camera and the panoramic image is an identity matrix, the x direction in the translation matrix is the resolution of the video picture, and the y direction in the translation matrix is 0; let n cameras be odd, n is P ₁,P₂,P₃…P_n respectively, the transformation matrix between the adjacent camera video pictures calculated is: h ₁,H₂,H₃…H_n-1, the subscript of the transformation matrix indicates the transformation relationship between the current camera i and the camera i+1 to the right of the current camera, and find the middle camera number as follows: (1+n)/2, assuming that P (1+n)/2 is a reference camera, the transformation from the reference camera to the panoramic image is a unitary matrix, and assuming that I is a translation matrix and T is a translation matrix, the transformation formula from the reference camera pixel point to the panoramic image is as follows:

P_p＝T*I*P_R

wherein,

Finally, each image can be mapped to the panorama by the formula;

The overlapping optimization unit is used for carrying out overlapping optimization on the overlapping areas of the two adjacent real-time video pictures in the panoramic image to obtain the panoramic image after the overlapping optimization processing, searching the overlapping areas of the adjacent images in the panoramic image, optimizing the overlapping areas of the adjacent images in the panoramic image, and generating the panoramic image, if the overlapping areas of the adjacent images in the panoramic image are not optimized, the overlapping areas in the panoramic image lose some details due to oversaturation of pixels, and the pixel calculation formula of the overlapping areas of the adjacent video pictures in the panoramic image is as follows:

P_(x,y+0)＝P_i,(x,y+0)*a+P_i+1,(x,y+0)*(1-z)

P_(x,y+1)＝P_i,(x,y+1)*a+P_i+1,(x,y+1)*(1-z)

P_(x,y+2)＝P_i,(x,y+2)*a+P_i+1,(x,y+2)*(1-z)

z＝((width*ratio)-(y-s)"/(width*ratio)

The three-dimensional fusion module is used for rendering the panoramic image to a corresponding position of a preset three-dimensional model in real time for display through a three-dimensional fusion technology, and comprises the following steps: a discrete point pair determining unit and a three-dimensional fusion unit;

A discrete point pair determining unit, configured to determine a three-dimensional model of a target area and a plurality of discrete point pairs of the panoramic image, where the discrete point pairs are formed by a three-dimensional model point coordinate and a panoramic image rasterized coordinate, select a static panoramic image as a reference for editing, respectively select a plurality of binary point pairs in a three-dimensional scene and a corresponding panoramic image, each point pair is formed by a three-dimensional scene point coordinate and a two-dimensional panoramic image rasterized coordinate, take a screen space as an interpolation space, interpolate three-dimensional selection points in the screen space, and convert the three-dimensional selection points in the three-dimensional space into two-dimensional points in the screen space through projective transformation;