CN105025287A - A Method for Constructing a Stereoscopic Panorama of a Scene Using Rotated Video Sequence Images - Google Patents

Abstract

The invention discloses a method for automatically generating an immersive stereo panoramic image by utilizing video sequence images of rotary shooting of a geographic scene. The method mainly comprises the steps that 1) characteristic matched pairs between successive frame images in the video sequence are extracted one by one, and internal and external parameters of a camera are estimated; 2) the virtual rotation speed and the strip size of the camera are calculated; 3) the strip position is calculated according an estimated parallax value in the principle of center circle projection; and 4) left and right strip images are extracted from video frame images according to the strip size and position, and panoramic stitching is carried out on the extracted left and right strip images to generate left and right panoramic images with parallax. The method has the advantages that the process of generating the stereo panoramic image is simple, the single digital camera carries out rotation shooting on the scene at the visual point which is approximately fixed to obtain the video sequence, and the left and right panoramic images with parallax can be automatically stitched, and real 3D observation can be carried out via different 3D display devices as red-green glasses.

Description

A Method for Constructing a Stereoscopic Panorama of a Scene Using Rotated Video Sequence Images

technical field

The invention relates to the fields of splicing technology of video images, computer vision, digital image processing, three-dimensional visualization of geographic information, etc., and in particular to a method for generating panoramic stereograms using video sequence images.

Background technique

As a common mass media, video sequence itself contains very rich information. It is not only easy to obtain, but also can express geographical space more vividly and objectively, forming a real scene of "seeing the world from the side". Based on video sequence images, research on the method of using video sequence images to generate panoramic 3D stereoscopic images will greatly enrich the capabilities of GIS in data acquisition, video browsing and modeling, and panoramic 3D stereoscopic images themselves are a kind of construction. The model expression method is not only intuitive, simple and convenient, but also can provide an excellent landscape expression method for 3DGIS.

Some scholars have carried out relevant research on the generation of panoramic stereoscopic images by splicing common video sequences. Ishiguro et al. used the method of equiangular rotation to splice the continuous shooting image sequences to generate panoramic stereoscopic images (Ishiguro H, Yamamoto M, Tsuji S. Omni- directional stereo[J]. IEEE Transaction on Pattern Analysis and Machine Intelligence, 1992,14(2):257-262). Based on the Ishiguro method, Peleg and Ben-Ezra used the circular projection method to simplify the imaging conditions of the panoramic stereo, without strict equiangular shooting, and initially realized the panoramic stereo imaging with ordinary cameras (Peleg S, Ben-Ezra Ezra M. Stereo panorama with a single camera [C]. IEEE Conference of Computer Vision and Pattern Recognition, 1999:395-401). JungukCho et al. used a single camera to rotate and shoot images, calculated the bands between overlapping images, and stitched the images to form a panoramic stereo image (Junguk C, Joon-Hyuk C, Yongmin T, et al. Stereo Panoramic Image Stitching with a Single Camera[C ]. IEEE International Conference on Consumer Electronics, 2013: 256-257). K ang et al. used the image layering method to calculate the depth information of the panoramic image, and on this basis generate a panoramic stereo image (Kang S B.Layered DepthPanoramas[C].IEEE Conference on Computer Vision and Pattern Recognition,2007:1-8 ).

The current panoramic stereo generation method based on image stitching has strict requirements on shooting conditions or more restrictions on shooting methods, and is not suitable for using ordinary digital cameras to randomly shoot video sequences of scenes to construct true three-dimensional stereo panoramas.

Contents of the invention

Based on the video sequence images, the invention uses a central circle projection model and adopts an adaptive strip size and position calculation method to splice the video sequence images to generate a stereoscopic panoramic image. This method makes full use of the redundant information between video sequence images to generate a true three-dimensional panoramic image, which can effectively avoid the inconvenience caused by using multiple cameras for shooting, and provides a convenient method for generating stereoscopic panoramic images for the general public.

In order to achieve the above object, the present invention adopts the following technical solutions:

The method for constructing a scene stereoscopic panorama by using the video sequence images shot by rotation comprises the following steps:

1) Read the video frame images one by one, extract feature matching pairs between consecutive images, and estimate the internal and external parameters of the camera;

2) Calculate the virtual speed of camera rotation by using the feature matching points between images, and determine the size of the strip according to the rotation speed of the camera;

3) Carry out cylindrical projection to the video image according to the central circle projection principle, and use the parallax estimation value to calculate the position of the image strip;

4) Extract the video frame images according to the size and position of the strips, extract the left strip images and the right strip images respectively, and perform panorama stitching on all the extracted left strip images and right strip images respectively to generate parallax images Left panorama image and right panorama image.

The concrete process of described step 1) is:

(1-1) Read continuous frame images in the video sequence one by one;

(1-2) Use the ORB (oriented FAST and Rotated BRIEF) algorithm to perform feature matching on video frame images;

(1-3) Due to the presence of noise and moving targets, there will be mismatching points in the matching set. According to the epipolar principle, the RANSAC (Random Sample Consensus) method is used to eliminate the mismatching points;

(1-4) Calculate the homography matrix transformation between images according to the correct matching point set, calculate the internal and external parameter matrix of the camera according to the homography transformation matrix, and use the beam adjustment principle to correct the internal and external parameter matrix of the camera.

The concrete process of described step 2) is:

(2-1) According to the matching feature point of the image frame extracted in step 1), calculate the horizontal offset and the vertical offset of every pair of feature points;

(2-2) The horizontal offsets of all feature point pairs are summed and averaged, and the horizontal rotation speed of the camera is calculated;

(2-3) The vertical offsets of all feature point pairs are summed and averaged, and the vertical rotation speed of the camera is calculated;

(2-4) On the basis of the first three steps, calculate the size of the stripe.

Described step 3) specific process is:

(3-1) Projecting the video sequence image onto the inner surface of the cylinder;

(3-2) A viewpoint circle with a diameter close to the base distance of the human eye is set inside the cylinder, and two virtual cameras, left and right, are constructed on the viewpoint circle to perform virtual imaging on the video images projected onto the cylinder surface;

(3-3) According to the strip size determined in step 2) of the virtual imaging and the rotation orientation and parallax estimation value of the virtual camera, the positions corresponding to the left and right virtual camera shooting strips are respectively calculated according to the central circle projection model.

Described step 4) specific process is:

(4-1) extract the left and right virtual camera strip images respectively from the cylindrical projection video image according to the calculated strip size and position;

(4-2) Perform panoramic splicing on the extracted left and right virtual camera image strips to obtain left and right panoramic images with parallax to form a stereoscopic panoramic image pair.

Compared with the prior art, the method of the present invention has the following characteristics:

1. The method of the present invention adopts a camera, and on the basis of central circle projection, adaptive strip extraction is performed on video sequence images to generate left and right panoramic images with parallax, which can effectively avoid using multiple cameras to shoot. inconvenient;

2. The generation process of the stereoscopic panorama of the present invention does not require manual intervention, has a high degree of automation, and can provide a convenient method for generating panoramic stereoscopic images for the general public.

Therefore, the stereoscopic panoramic image generation method of the present invention is simple, only needs one camera to rotate and shoot the scene, and is suitable for ordinary users to use general video equipment to generate stereoscopic panoramic images from a video sequence that is surrounded by approximately fixed viewpoints. True three-dimensional stereo observation can be performed on true three-dimensional display devices (such as red and green glasses).

Description of drawings

Fig. 1 is the flowchart of the embodiment of the present invention;

The video image cylindrical projection of Fig. 2 embodiment of the present invention;

The central circle projection principle of the embodiment of the present invention in Fig. 3;

The simplified calculation model of the central circle projection of the embodiment of the present invention in Fig. 4;

Fig. 5 shows the cylindrical projection and left and right strip image extraction of the video sequence of the present invention.

Detailed ways

Further detailed description will be given below in conjunction with the accompanying drawings and embodiments.

As shown in Figure 1, the method of constructing a stereoscopic panorama of a scene using the video sequence images shot by rotation includes the following four parts:

Step 1: Read the video frame images one by one, extract feature matching pairs between consecutive images, and estimate the internal and external parameters of the camera;

Step 2 calculates the virtual speed of the camera rotation by using the feature matching points between the images, and calculates the band size according to the rotation speed of the camera;

Step 3: Perform cylindrical projection on the video image according to the central circle projection principle, and calculate the position of the image strip by using the estimated value of parallax;

Step 4 Extract the strips from the video frame image according to the size and position of the strips, extract the left strip image and the right strip image respectively, and splicing all the extracted left strip images and right strip images respectively to generate The left panoramic image and the right panoramic image, and use the observation equipment corresponding to the display mode (taking red and green glasses as an example) for immersive three-dimensional observation and experience;

The specific implementation steps of this embodiment are as follows:

Step 1 Read the video frame images one by one, extract feature matching pairs between consecutive images, and estimate the internal and external parameters of the camera:

(1-1) Read the continuous frame images in the video sequence one by one, and use the ORB algorithm to perform feature matching on the video frame images. In order to further refine the correctness of the matching point pairs between the images, the symmetrical matching principle is used to match;

(1-2) Due to the existence of noise and moving targets, there will be mismatching points in the matching set. According to the principle of epipolar lines, two corresponding key points are distributed on their respective epipolar lines. The expression is as follows:

${\mathrm{<span\; class="notranslate">\; u</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}^{\mathrm{<span\; class="notranslate">\; T</span>}}{\mathrm{<span\; class="notranslate">\; f</span>}}_{\mathrm{<span\; class="notranslate">\; 12</span>}}{\mathrm{<span\; class="notranslate">\; u</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 0</span>}<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; )</span>$

Among them, u ₁ and u ₂ are the coordinates of the matching points corresponding to the two images; F ₁₂ is the fundamental matrix; according to formula (1), the RANSAC method is used to eliminate the wrong matching points.

(1-3) Calculate the homography transformation matrix between images according to the correct matching point set, calculate the internal and external parameter matrix of the camera according to the homography transformation matrix, and correct the internal and external parameter matrix of the camera according to the beam adjustment principle.

(1) Image homography matrix transformation

n(>4) pairs of matching points of two images The homography matrix can be calculated in the sense of differing by a constant factor, the algorithm is as follows:

make $h=\left[\begin{array}{ccc}{h}_{01}& h_{02}& h_{03}\\ h_{11}& h_{12}& h_{13}\\ h_{\mathrm{twenty\; one}}& h_{\mathrm{twenty\; two}}& h_{\mathrm{twenty\; three}}\end{array}\right],$ Its vector form: h=(h ₀₁ , h ₀₂ , h ₀₃ , h ₁₁ , h ₁₂ , h ₁₃ , h ₂₁ , h ₂₂ , h ₂₃ ) ^T . For matching point pair m=(x,y,1) ^T , m'=(x',y',1) ^T , from The following two linear equations about h can be obtained:

(x,y,1,0,0,0,x'x,x'y)h=x' (2)

(0,0,0,x,y,1,y'x,y'y)h=y'

$\begin{array}{cc}{\mathrm{<span\; class="notranslate">\; x</span>}}^{<span\; class="notranslate">\; \&prime;</span>}<span\; class="notranslate">\; =</span>\frac{{\mathrm{<span\; class="notranslate">\; h</span>}}_{\mathrm{<span\; class="notranslate">\; 00</span>}}\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; h</span>}}_{\mathrm{<span\; class="notranslate">\; 01</span>}}\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; h</span>}}_{\mathrm{<span\; class="notranslate">\; 02</span>}}}{{\mathrm{<span\; class="notranslate">\; h</span>}}_{\mathrm{<span\; class="notranslate">\; 20</span>}}\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; h</span>}}_{\mathrm{<span\; class="notranslate">\; twenty\; one</span>}}\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; h</span>}}_{\mathrm{<span\; class="notranslate">\; twenty\; two</span>}}}& {\mathrm{<span\; class="notranslate">\; the\; y</span>}}^{<span\; class="notranslate">\; \&prime;</span>}<span\; class="notranslate">\; =</span>\frac{{\mathrm{<span\; class="notranslate">\; h</span>}}_{\mathrm{<span\; class="notranslate">\; 10</span>}}\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; h</span>}}_{\mathrm{<span\; class="notranslate">\; 11</span>}}\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; h</span>}}_{\mathrm{<span\; class="notranslate">\; 12</span>}}}{{\mathrm{<span\; class="notranslate">\; h</span>}}_{\mathrm{<span\; class="notranslate">\; 20</span>}}\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; h</span>}}_{\mathrm{<span\; class="notranslate">\; twenty\; one</span>}}\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; h</span>}}_{\mathrm{<span\; class="notranslate">\; twenty\; two</span>}}}\end{array}<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 3</span>}<span\; class="notranslate">\; )</span>$

In this way, the homography matrix H can be obtained by selecting more than 4 pairs of matching points.

(2) Camera focal length estimation

First calculate the homography matrix H _ij between consecutive image pairs in the video sequence, according to the definition of the plane homography matrix, H=K(R+tn ^T )K ^-1 , there is:

${\mathrm{<span\; class="notranslate">\; h</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; \&cong;</span>{\mathrm{<span\; class="notranslate">\; K</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}{\mathrm{<span\; class="notranslate">\; R</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}\mathrm{<span\; class="notranslate">\; j</span>}}{\mathrm{<span\; class="notranslate">\; K</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}^{<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 4</span>}<span\; class="notranslate">\; )</span>$

Rewrite formula (4) as:

${\mathrm{<span\; class="notranslate">\; R</span>}}_{\mathrm{<span\; class="notranslate">\; 10</span>}}<span\; class="notranslate">\; ~</span>{\mathrm{<span\; class="notranslate">\; K</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}^{<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; h</span>}}_{\mathrm{<span\; class="notranslate">\; 10</span>}}{\mathrm{<span\; class="notranslate">\; K</span>}}_{\mathrm{<span\; class="notranslate">\; 0</span>}}<span\; class="notranslate">\; ~</span>\left[\begin{array}{ccc}{\mathrm{<span\; class="notranslate">\; h</span>}}_{\mathrm{<span\; class="notranslate">\; 00</span>}}& {\mathrm{<span\; class="notranslate">\; h</span>}}_{\mathrm{<span\; class="notranslate">\; 01</span>}}& {\mathrm{<span\; class="notranslate">\; f</span>}}_{\mathrm{<span\; class="notranslate">\; 0</span>}}^{<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; h</span>}}_{\mathrm{<span\; class="notranslate">\; 02</span>}}\\ {\mathrm{<span\; class="notranslate">\; h</span>}}_{\mathrm{<span\; class="notranslate">\; 10</span>}}& {\mathrm{<span\; class="notranslate">\; h</span>}}_{\mathrm{<span\; class="notranslate">\; 11</span>}}& {\mathrm{<span\; class="notranslate">\; f</span>}}_{\mathrm{<span\; class="notranslate">\; 0</span>}}^{<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; h</span>}}_{\mathrm{<span\; class="notranslate">\; 12</span>}}\\ {\mathrm{<span\; class="notranslate">\; f</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; h</span>}}_{\mathrm{<span\; class="notranslate">\; 20</span>}}& {\mathrm{<span\; class="notranslate">\; f</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; h</span>}}_{\mathrm{<span\; class="notranslate">\; twenty\; one</span>}}& {\mathrm{<span\; class="notranslate">\; f</span>}}_{\mathrm{<span\; class="notranslate">\; 0</span>}}^{<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; f</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; h</span>}}_{\mathrm{<span\; class="notranslate">\; twenty\; two</span>}}\end{array}\right]<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 5</span>}<span\; class="notranslate">\; )</span>$

in, $h_{10}=\left[\begin{array}{ccc}{h}_{00}& h_{01}& h_{02}\\ h_{10}& h_{11}& h_{12}\\ h_{20}& h_{\mathrm{twenty\; one}}& h_{\mathrm{twenty\; two}}\end{array}\right],$ Taking advantage of the orthogonality of the rotation matrix _R10 ,

can get:

${\mathrm{<span\; class="notranslate">\; h</span>}}_{\mathrm{<span\; class="notranslate">\; 00</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; h</span>}}_{\mathrm{<span\; class="notranslate">\; 01</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; f</span>}}_{\mathrm{<span\; class="notranslate">\; 0</span>}}^{<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 2</span>}}{\mathrm{<span\; class="notranslate">\; h</span>}}_{\mathrm{<span\; class="notranslate">\; 02</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; =</span>{\mathrm{<span\; class="notranslate">\; h</span>}}_{\mathrm{<span\; class="notranslate">\; 10</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; h</span>}}_{\mathrm{<span\; class="notranslate">\; 11</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; f</span>}}_{\mathrm{<span\; class="notranslate">\; 0</span>}}^{<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 2</span>}}{\mathrm{<span\; class="notranslate">\; h</span>}}_{\mathrm{<span\; class="notranslate">\; 12</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 6</span>}<span\; class="notranslate">\; )</span>$

${\mathrm{<span\; class="notranslate">\; h</span>}}_{\mathrm{<span\; class="notranslate">\; 00</span>}}{\mathrm{<span\; class="notranslate">\; h</span>}}_{\mathrm{<span\; class="notranslate">\; 10</span>}}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; h</span>}}_{\mathrm{<span\; class="notranslate">\; 01</span>}}{\mathrm{<span\; class="notranslate">\; h</span>}}_{\mathrm{<span\; class="notranslate">\; 11</span>}}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; f</span>}}_{\mathrm{<span\; class="notranslate">\; 0</span>}}^{<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 2</span>}}{\mathrm{<span\; class="notranslate">\; h</span>}}_{\mathrm{<span\; class="notranslate">\; 02</span>}}{\mathrm{<span\; class="notranslate">\; h</span>}}_{\mathrm{<span\; class="notranslate">\; 12</span>}}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 0</span>}<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 7</span>}<span\; class="notranslate">\; )</span>$

From (6) and (7) two formulas can be calculated:

$f_0^2=\frac{h_{12}^2-h_{02}^2}{h_{00}^2+h_{01}^2-h_{10}^2-h_{11}^2}$ and $h_{00}^2+h_{01}^2\&NotEqual;h_{10}^2+h_{11}^2---\left(8\right)$

or: $f_0^2=-\frac{h_{01}{h}_{12}}{h_{00}{h}_{10}+h_{01}{h}_{11}}$ And h ₀₀ h ₁₀ ≠ -h ₀₁ h ₁₁ (9)

f ₁ is also obtained in a similar way. Since this paper assumes that the focal length of the camera remains unchanged during the shooting process, the final estimate of the camera focal length f can be calculated by the geometric mean of f ₀ and f ₁ , In the panoramic image sequence, the respective focal length estimates are obtained from different homography matrices, and the final f should take the geometric mean of all focal lengths. Finally, the rotation transformation matrix R between the two images is calculated according to the obtained internal reference matrix K.

Step 2 Calculate the virtual speed of camera rotation according to the feature matching points between images, and calculate the strip size according to the rotation speed of the camera:

(2-1) Calculate the virtual speed of camera rotation according to the number of matching point pairs detected in two consecutive frames of images:

${\mathrm{<span\; class="notranslate">\; V</span>}}_{\mathrm{<span\; class="notranslate">\; h</span>}}<span\; class="notranslate">\; =</span>\frac{\underset{\mathrm{<span\; class="notranslate">\; i</span>}}{\overset{\mathrm{<span\; class="notranslate">\; no</span>}}{<span\; class="notranslate">\; \&Sigma;</span>}}{\mathrm{<span\; class="notranslate">\; p</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; p</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; )</span>}{\mathrm{<span\; class="notranslate">\; no</span>}}<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 10</span>}<span\; class="notranslate">\; )</span>$

${\mathrm{<span\; class="notranslate">\; V</span>}}_{\mathrm{<span\; class="notranslate">\; p</span>}}<span\; class="notranslate">\; =</span>\frac{\underset{\mathrm{<span\; class="notranslate">\; i</span>}}{\overset{\mathrm{<span\; class="notranslate">\; no</span>}}{<span\; class="notranslate">\; \&Sigma;</span>}}{\mathrm{<span\; class="notranslate">\; p</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; p</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; )</span>}{\mathrm{<span\; class="notranslate">\; no</span>}}<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 11</span>}<span\; class="notranslate">\; )</span>$

Among them, V _h is the virtual velocity in the horizontal direction, V _p is the virtual velocity in the vertical direction, p _1i (x) represents the abscissa x of the i-th matching point in the frame 1 image, and p _2i (x) represents the The abscissa x of the i-th matching point, p _1i (y) represents the ordinate y of the i-th matching point in the frame 1 image, and p _2i (y) represents the ordinate y of the i-th matching point in the frame 2 image.

(2-2) Calculate the stripe size △D according to the rotation speed of the camera:

△D＝V _h (12)

Step 3: Carry out cylindrical projection on the video image according to the central circle projection principle, and calculate the position of the image strip by using the estimated value of disparity:

(3-1) Carry out cylindrical projection on the video image and project it onto the inner surface of the cylinder:

The point on the cylinder is determined by the parameterization of angle θ and height h, as shown in Figure 2, and the corresponding relationship is as follows:

(sinθ,h,cosθ)∝(x,y,f) (13)

According to the corresponding relationship in (13), the coordinate form mapped from the image plane to the cylindrical surface can be calculated:

${\mathrm{<span\; class="notranslate">\; x</span>}}^{<span\; class="notranslate">\; \&prime;</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; the\; s</span>}\mathrm{<span\; class="notranslate">\; \&theta;</span>}<span\; class="notranslate">\; =</span>{\mathrm{<span\; class="notranslate">\; stan</span>}}^{<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; (</span>\frac{\mathrm{<span\; class="notranslate">\; x</span>}}{\mathrm{<span\; class="notranslate">\; f</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 14</span>}<span\; class="notranslate">\; )</span>$

${\mathrm{<span\; class="notranslate">\; the\; y</span>}}^{<span\; class="notranslate">\; \&prime;</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; the\; s</span>}\mathrm{<span\; class="notranslate">\; h</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; the\; s</span>}\frac{\mathrm{<span\; class="notranslate">\; the\; y</span>}}{\sqrt{{\mathrm{<span\; class="notranslate">\; x</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; the\; y</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}}}<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 15</span>}<span\; class="notranslate">\; )</span>$

Among them, (x, y) is the plane image coordinates, (x', y') is the cylindrical coordinates, s is the radius of the cylinder, generally s=f to minimize the deformation degree of the image center.

Since the cylinder is a developable surface, the image can be translated and rotated in cylindrical coordinates to maintain the shape. The inverse mapping formula is expressed as follows:

$\begin{array}{c}\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; f</span>}\mathrm{<span\; class="notranslate">\; t</span>}\mathrm{<span\; class="notranslate">\; a</span>}\mathrm{<span\; class="notranslate">\; no</span>}\mathrm{<span\; class="notranslate">\; \&theta;</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; f</span>}\mathrm{<span\; class="notranslate">\; t</span>}\mathrm{<span\; class="notranslate">\; a</span>}\mathrm{<span\; class="notranslate">\; no</span>}\frac{{\mathrm{<span\; class="notranslate">\; x</span>}}^{<span\; class="notranslate">\; \&prime;</span>}}{\mathrm{<span\; class="notranslate">\; the\; s</span>}}\\ \mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; h</span>}\sqrt{{\mathrm{<span\; class="notranslate">\; x</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; f</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}}<span\; class="notranslate">\; =</span>\frac{{\mathrm{<span\; class="notranslate">\; the\; y</span>}}^{<span\; class="notranslate">\; \&prime;</span>}}{\mathrm{<span\; class="notranslate">\; the\; s</span>}}\mathrm{<span\; class="notranslate">\; f</span>}\sqrt{\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; the\; tan</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}\frac{{\mathrm{<span\; class="notranslate">\; x</span>}}^{<span\; class="notranslate">\; \&prime;</span>}}{\mathrm{<span\; class="notranslate">\; the\; s</span>}}}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; f</span>}\frac{{\mathrm{<span\; class="notranslate">\; the\; y</span>}}^{<span\; class="notranslate">\; \&prime;</span>}}{\mathrm{<span\; class="notranslate">\; the\; s</span>}}\mathrm{<span\; class="notranslate">\; sec</span>}\frac{{\mathrm{<span\; class="notranslate">\; x</span>}}^{<span\; class="notranslate">\; \&prime;</span>}}{\mathrm{<span\; class="notranslate">\; the\; s</span>}}\end{array}<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 16</span>}<span\; class="notranslate">\; )</span>$

(3-2) In order to realize stereoscopic vision, according to the central circle projection principle, as shown in Figure 3, a central circle with a diameter twice the base distance of the human eye is set up in the cylinder, and a virtual center circle is placed on the left and right sides of the central circle. Cameras V _L , _VR , the distance is recorded as 2d. The real camera is placed at the optical center O, and the radius of the rotation axis is r. The camera rotates 360 degrees around the rotation axis for continuous video shooting. The captured images are projected onto the cylindrical projection surface, and the light rays of the virtual camera are projected along the optical center O to Cylindrical projection surface.

(3-3) In order to facilitate the calculation of the strip position, the central circle projection model is simplified, as shown in Figure 4, the difference between the left and right virtual cameras V _R and V _L is 2d, the radius of rotation is denoted as r, and the focal length of the camera Estimated from step 1, denoted as f, and the difference between the left and right strip positions is denoted as 2v. According to the principle of similar triangles, the relationship between strips and parallax can be calculated (see Figure 4):

$\frac{\mathrm{<span\; class="notranslate">\; 2</span>}\mathrm{<span\; class="notranslate">\; v</span>}}{\mathrm{<span\; class="notranslate">\; 2</span>}\mathrm{<span\; class="notranslate">\; d</span>}}<span\; class="notranslate">\; =</span>\frac{\mathrm{<span\; class="notranslate">\; f</span>}}{\mathrm{<span\; class="notranslate">\; r</span>}}<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 17</span>}<span\; class="notranslate">\; )</span>$

In the formula, d is the set human eye base distance, r is the average value of the rotation radius when the camera rotates and shoots, and f is the internal reference of the camera calculated in step 1. After v is obtained according to the proportional formula, the corresponding position of the strip can be calculated.

Step 4 Extract the strips from the video frame image according to the calculated strip size and position, extract the left strip image and the right strip image respectively, and splicing all the extracted left strip images and right strip images respectively to generate The left panoramic image and the right panoramic image with parallax, and using the observation equipment corresponding to the display mode (taking red and green glasses as an example), can observe the immersive panoramic stereoscopic image. The specific steps are as follows:

(4-1) For the video sequence, there are many images generated, and it is necessary to perform cylindrical projection on all the images of each frame according to the central circle method, and obtain the projected video sequence as shown in FIG. 5 .

(4-2) strip extraction is carried out to the video frame image according to the strip size and position obtained by calculation, respectively extracting the left strip image and the right strip image (see Fig. 5);

(4-3) Panoramic splicing is carried out respectively to all extracted left and right strip images to generate a left panorama and a right panorama with parallax;

(4-4) adopt the red-green synthesis method respectively to the left and right panorama images to carry out the red-green synthesis to the panorama image to generate the red-green three-dimensional panorama image, and the synthesis method is as follows:

The process of processing the left panoramic image is as follows:

(1) Select a magenta mask image with the same size as the left panoramic image, and the RGB value of each pixel of the mask image is (255,0,0).

(2) Each pixel of the left panoramic image is ORed with each pixel of the mask image to obtain a new left panoramic image.

The calculation method is as follows:

Let the color RGB of a pixel (i, j) in the left panoramic image be (r, g, b), and the color RGB of the same position (i, j) in the red mask image be (255, 0, 0), then Perform "or" operation, and the color RGB of the new pixel is (255, g, b).

Similarly, the same method can be used for the right panoramic image to perform "OR" operation with the green mask image (RGB(0,255,0)) to obtain a new right panoramic image.

After the new left panorama image A and the new right panorama image B are generated, it is necessary to combine the two panorama images A and B into a red-green panorama image to display the stereoscopic effect. Images are ANDed.

Set the RGB value (255, g, b) of a pixel (i, j) in the A panoramic image, and the RGB value (r, 255, b) of the corresponding pixel in the B panoramic image, and perform "AND" operation to obtain a new The RGB values are (r',g',b'). Perform an "AND" operation on all the pixels in the A and B images to obtain a new composite red-green panoramic image.

Claims (5)

1. The method for constructing the scene three-dimensional panorama by using the video sequence images shot in a rotating way is characterized by comprising the following steps:

1) reading video frame images one by one, extracting feature matching pairs between continuous images, and estimating internal and external parameters of a camera;

2) calculating a virtual speed of the rotation of the camera by using the feature matching points between the images, and determining the size of the strip according to the rotation speed of the camera;

3) performing cylindrical projection on the video image according to a central circle projection principle, and calculating the position of an image strip by using a parallax estimation value;

4) and extracting the video frame images according to the sizes and the positions of the strips, respectively extracting a left strip image and a right strip image, and respectively carrying out panoramic stitching on all the extracted left strip images and right strip images to generate a left panoramic image and a right panoramic image with parallax.

2. The method for constructing a stereoscopic panorama of a scene using images of a video sequence captured by rotation according to claim 1, wherein the specific process of step 1) is as follows:

(1-1) reading continuous frame images in a video sequence one by one;

(1-2) performing feature matching on the video frame image by adopting an ORB algorithm;

(1-3) due to the existence of noise and a moving target, mismatching points exist in the matching set, and the mismatching points are removed by adopting a RANSAC method according to an epipolar line principle;

and (1-4) calculating homography matrix transformation between images according to the correct matching point set, calculating an internal and external parameter matrix of the camera according to the homography transformation matrix, and correcting the internal and external parameter matrix of the camera by using a light beam adjustment principle.

3. The method for constructing a stereoscopic panorama of a scene using images of a video sequence captured by rotation according to claim 1, wherein the specific process of step 2) is:

(2-1) calculating the horizontal offset and the vertical offset of each pair of feature points according to the matched feature points of the image frame extracted in the step 1);

(2-2) summing and averaging the horizontal offset of all characteristic point pairs, and calculating the horizontal rotation speed of the camera;

(2-3) summing and averaging the vertical offsets of all the characteristic point pairs, and calculating the vertical rotation speed of the camera;

and (2-4) calculating the size of the strip on the basis of the previous three steps.

4. The method for constructing a stereoscopic panorama of a scene using images of a video sequence captured by rotation according to claim 1, wherein the specific process of the step 3) is as follows:

(3-1) projecting the video sequence images onto the inner surface of the cylinder;

(3-2) arranging a viewpoint circle with the diameter similar to the base distance of human eyes in the cylinder, constructing a left virtual camera and a right virtual camera on the viewpoint circle, and virtually imaging the video image projected to the cylindrical surface;

and (3-3) respectively calculating the corresponding positions of the shooting strips of the left virtual camera and the right virtual camera according to the size of the strips determined in the step 2) in the virtual imaging and the rotation direction and the parallax estimation value of the virtual camera and a central circle projection model.

5. The method for constructing a stereoscopic panorama of a scene using images of a video sequence captured by rotation as claimed in claim 4, wherein the specific process of the step 4) is as follows:

(4-1) respectively extracting left and right virtual camera strip images from the cylindrical projection video image according to the calculated strip size and position;

and (4-2) carrying out panoramic stitching on the extracted left and right virtual camera image strips to obtain left and right panoramic images with parallax, so as to form a stereo panoramic image pair.