CN104778658A - Full-automatic geometric mosaic correction method for images projected by multiple projectors - Google Patents

Abstract

The present invention provides a method for automatic geometric splicing and correction of multi-projector projection images. The specific process is as follows: step 1, calibrate the pose [R _c | T _c ] of the camera relative to the screen; step 2, calibrate the relative position of each projector based on the pose of the screen [R _p |T _p ]; step 3, point X _wnew on the screen to the corresponding pixel x _r on the right projector, using the mapping relationship between each projector pixel and camera pixel, It is mapped to the left projector to obtain the pixel point x _lnew corresponding to X _wnew in the left projector; step 4, according to the corresponding points x _p and X _w and the corresponding points x _lnew and X _wnew , for each left projector two The second calibration updates its pose relative to the screen [R _p |T _p ]; step 5, using the pose of each projector relative to the screen [R _p |T _p ] to generate the projection image of each projector, using the optimization Methods The projected image is geometrically deformed, and the deformed corrected image is used as the projected image of each projector. This method can realize splicing correction of non-planar screen projection images.

Description

A method for automatic geometric mosaic correction of multi-projector projected images

technical field

The invention relates to a method for automatic geometric mosaic correction of multi-projector projected images, which is suitable for high immersion display, digital simulation, virtual reality and other technical fields.

Background technique

In virtual reality and digital media, non-planar projections are often used to create highly immersive, high-resolution display environments. For this kind of non-planar screen projection, the following two problems will be faced: first, the image directly projected onto the non-planar surface will be geometrically deformed, and the image needs to be pre-distorted to adapt to the shape of the screen; secondly, in order to improve the display resolution or enlarge the The dynamic range of brightness often needs to be projected by multiple projectors at the same time, and the geometric texture alignment and brightness fusion of overlapping areas between different projectors have become unavoidable problems.

In order to solve the above problems, the existing projection correction methods start from two aspects: one is the three-dimensional reconstruction method, which restores the geometric shape of the projection scene, and at the same time obtains the relative position of the projector in the scene, so as to realize projection image correction ; the other is a non-reconstruction method that uses a purely 2D image transformation method to achieve geometric texture alignment of projected images on non-planar surfaces. The advantage of the 3D reconstruction method is that all scene information can be obtained, so it is more convenient to generate the corresponding position projector image, which is especially suitable for the immersive projection environment where the user's viewpoint is not fixed. The disadvantage is that the equipment used for reconstruction is expensive and the process is complicated. Difficult to achieve high precision. Non-reconstruction methods often use methods such as homography or Bezier deformation to adjust images, and use human eyes or cameras as the basis for texture alignment judgments. In planar projection, it is necessary to obtain the corresponding relationship between the projector and the screen by using auxiliary marker points or other means, and the calibration process requires a lot of manual participation.

Contents of the invention

In view of this, the purpose of the present invention is to solve the problem of geometric deformation of multi-projector projected images when non-planar screen projection occurs, and propose a multi-projector projected image automatic geometric mosaic correction method.

Realize the technical scheme of the present invention as follows:

A method for automatic geometric mosaic correction of multi-projector projected images, the specific process is:

Step 1. Set up a camera in front of the screen, and use the corresponding relationship between the spatial point X _w on the screen and the camera pixel point x _c to calibrate the pose [R _c | T _c ] of the camera relative to the screen;

Step 2: Establish the mapping relationship between each projector pixel point x _p and camera pixel point x _c , based on the pose [R _c | T _c ] and the screen shape parameters, obtain the space point X _w on the screen and the projector The corresponding relationship of pixel x _p , calibrate the pose of each projector relative to the screen [R _p |T _p ];

Step 3. For any two projectors whose projection areas overlap, define the projector corresponding to the right projection area in the overlapping projection area as the right projector, and define the projector corresponding to the left projection area as the left projector; The corresponding pixel x _r of the upper point X _wnew on the right projector is mapped to the left projector by using the mapping relationship between each projector pixel and the camera pixel, and the corresponding X _wnew in the left projector is obtained pixel x _lnew ;

Step 4: For the left projector among any two projectors whose projected images overlap, according to the corresponding points x _p and X _w and the corresponding points x _lnew and X _wnew , update the relative The pose on the screen [R _p |T _p ];

Step 5: Use the position and orientation of each projector relative to the screen [R _p |T _p ] to generate the projection image of each projector, use the optimization method shown in the following formula to geometrically deform the projection image, and correct the deformed The image is used as the projected image of each projector;

$\left\{\begin{array}{cc}\mathrm{<span\; class="notranslate">\; min</span>}& \mathrm{<span\; class="notranslate">\; f</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; p</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\frac{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; N</span>}}\underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; N</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}{<span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; idst</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; iB</span>}}<span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>}_{\mathrm{<span\; class="notranslate">\; 2</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}\\ \mathrm{<span\; class="notranslate">\; the\; s</span>}<span\; class="notranslate">\; .</span>\mathrm{<span\; class="notranslate">\; t</span>}<span\; class="notranslate">\; .</span>& {<span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; p</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; p</span>}}_{\mathrm{<span\; class="notranslate">\; 0</span>}\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; \&le;</span>\mathrm{<span\; class="notranslate">\; w</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; p</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; )</span>\end{array}\right.$

Among them, it is assumed that there are N coincident points between the images projected by two adjacent projectors, x _idst represents the target position of the i-th coincident point, and x _iB represents the position of the i-th coincident point after Bessel optimization , the objective function f(p) represents the average stitching error of all coincident points, p _j represents the current position of the jth Bezier control point, p _0j represents the initial position of the jth Bezier control point, w(p _j ) represents the moving range of p _j .

Beneficial effect

Firstly, the present invention adopts the whole process of geometric calibration without 3D reconstruction, uses the camera with calibrated internal parameters as the measurement medium to establish the corresponding relationship between the projector and the screen, and uses the parameterized screen shape to realize the pose calibration of the camera and projector.

Second, the present invention performs constrained calibration on two projectors whose projection images overlap, and makes full use of the information of each projector to make the pixel matching error in the fusion band of different projectors uniform.

Thirdly, the present invention uses the image Bezier geometric adjustment method to realize the image geometric texture alignment in the overlapping area, and realizes automatic and fast multi-projector geometric correction.

Description of drawings

FIG. 1 is a flow chart of the method for automatic geometric mosaic correction of multi-projector projected images according to the present invention.

Fig. 2 is a schematic diagram of the imaging relationship among the screen, the projector and the camera.

FIG. 3 is a schematic diagram of establishing a pixel mapping relationship between a projector and a camera through structured light mapping and initially calibrating the projection matrix of the projector.

FIG. 4 is a schematic diagram of projector constrained calibration using adjacent projectors.

Fig. 5 is a schematic diagram of using Bezier interpolation and optimization methods to realize geometric adjustment of projected images to meet texture alignment in overlapping regions.

Figure 6 takes three projectors as an example, a schematic diagram of texture overlap before calibration and after calibration.

Detailed ways

The present invention will be described in detail below in conjunction with the accompanying drawings and specific examples.

In this embodiment, a camera is used as a measurement medium, a non-reconstruction method is used to calibrate the relative position of the projector and the screen, and an optimization method is used to globally adjust the projected image to meet the projection correction method for image alignment in overlapping areas; Taking the common cylindrical projection environment as an example, the multi-projector non-planar automatic geometric correction algorithm includes the following steps:

(1) Camera pose estimation:

As shown in Figure 2, a camera is set in front of the column screen, and a few external dimension parameters such as the length, width, and height of the column screen are measured to obtain the column screen shape parameters; using the corresponding relationship between the spatial point X _w on the column screen and the camera pixel point x _c , Using the imaging formula, estimate the pose [R _c |T _c ] of the calibration camera relative to the column curtain, as shown in formula (1).

$\stackrel{<span\; class="notranslate">\; ~</span>}{{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; c</span>}}}<span\; class="notranslate">\; =</span>{\mathrm{<span\; class="notranslate">\; P</span>}}_{\mathrm{<span\; class="notranslate">\; c</span>}}\stackrel{<span\; class="notranslate">\; ~</span>}{{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; w</span>}}}<span\; class="notranslate">\; =</span>{\mathrm{<span\; class="notranslate">\; K</span>}}_{\mathrm{<span\; class="notranslate">\; c</span>}}<span\; class="notranslate">\; [</span>{\mathrm{<span\; class="notranslate">\; R</span>}}_{\mathrm{<span\; class="notranslate">\; c</span>}}<span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; T</span>}}_{\mathrm{<span\; class="notranslate">\; c</span>}}<span\; class="notranslate">\; ]</span>\stackrel{<span\; class="notranslate">\; ~</span>}{{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; w</span>}}}<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; )</span>$

Among them, K _c is the internal reference matrix of the camera, which can be obtained by calibrating the camera, R _c is the rotation matrix of the camera, T _c is the translation matrix of the camera, P _c =K _c [R _c |T _c ], $\widetilde{x_c}=\left[\begin{array}{c}u\\ v\\ 1\end{array}\right],$ (u, v) represents the pixel coordinates on the projector plane, $\widetilde{x_w}=\left[\begin{array}{c}x\\ \mathrm{the\; y}\\ z\\ 1\end{array}\right],$ (x, y, z) represent the three-dimensional coordinates of the space point on the column curtain.

(2) Perform projector projection matrix calibration:

As shown in Figure 3, the mapping relationship x _c = f(x _p ) between each projector pixel point x _p and camera pixel point x _c is established using structured light coding; _c | T _c ] and the shape parameters of the column screen, the corresponding relationship between the space point X _w on the column screen and the pixel point x _p of the projector is obtained, as shown in formula (2);

X _w ＝f _b (x _c )＝f _b (f(x _p )) (2)

Use equation (3) to initially calibrate the projector pose [R _p |T _p ], as shown in Figure 2.

$\stackrel{<span\; class="notranslate">\; ~</span>}{{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; p</span>}}}<span\; class="notranslate">\; =</span>{\mathrm{<span\; class="notranslate">\; P</span>}}_{\mathrm{<span\; class="notranslate">\; p</span>}}\stackrel{<span\; class="notranslate">\; ~</span>}{{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; w</span>}}}<span\; class="notranslate">\; =</span>{\mathrm{<span\; class="notranslate">\; K</span>}}_{\mathrm{<span\; class="notranslate">\; p</span>}}<span\; class="notranslate">\; [</span>{\mathrm{<span\; class="notranslate">\; R</span>}}_{\mathrm{<span\; class="notranslate">\; p</span>}}<span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; T</span>}}_{\mathrm{<span\; class="notranslate">\; p</span>}}<span\; class="notranslate">\; ]</span>\stackrel{<span\; class="notranslate">\; ~</span>}{{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; w</span>}}}<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 3</span>}<span\; class="notranslate">\; )</span>$

in, $x_w=\left[\begin{array}{c}x\\ \mathrm{the\; y}\\ z\end{array}\right],x_c=\left[\begin{array}{c}{u}_c\\ v_c\end{array}\right],x_p=\left[\begin{array}{c}{u}_p\\ v_p\end{array}\right],\widetilde{x_p}=\left[\begin{array}{c}{u}_p\\ v_p\\ 1\end{array}\right],$ K _p is the internal reference matrix of the projector.

(3) Carry out projector constraint calibration:

In order to reduce the stitching error of the projector, it is necessary to calibrate the pose of the projector twice. The specific process is as follows:

As shown in Figure 4, for any two projectors whose projection areas overlap, the projector corresponding to the right projection area in the overlapping projection area is defined as the right projector, and the projector corresponding to the left projection area is defined as the left projector ; Use the pixel mapping relationship between the projector and the camera established by structured light mapping x _c =f(x _p ), use the following formula to map the corresponding point x _r of a point X _wnew on the screen to the left projector In the projector, get the new corresponding point x _lnew of X _wnew in the left projector

x _lnew ＝f _l2c (x _c )＝f _l2c (f _c2r (x _r )) (4) Among them, f _A2B (x) represents the structured light mapping relationship between A and B, for example, f _c2r (x _r ) represents The structured light mapping relationship between the camera and the right projector.

Use the new corresponding points x _lnew and X _wnew and the corresponding points x _p and X _w used in the initial calibration to use (3) to perform secondary calibration on the left projector to obtain the pose of the left projector under the constraints of the right projector, In this way, the calculation errors of the poses of each projector are evened out.

(4) After constraint calibration, due to the projection matrix calibration error and the influence of the flatness of the cylindrical screen surface, the projector splicing error is still relatively large, and it is necessary to use the Bessel interpolation optimization method to further reduce the splicing error; therefore, this step first uses each projector position pose, generate the projection images of each projector, and then use the optimization method (as shown in Equation 5) to geometrically deform the projection images according to the ideal image under the camera angle of view, obtain the corrected image after deformation, and realize the projection under the camera angle of view The geometric texture alignment of the machine overlapping area, as shown in Figure 5;

$\left\{\begin{array}{cc}\mathrm{<span\; class="notranslate">\; min</span>}& \mathrm{<span\; class="notranslate">\; f</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; p</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\frac{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; N</span>}}\underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; N</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}{<span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; idst</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; iB</span>}}<span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>}_{\mathrm{<span\; class="notranslate">\; 2</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}\\ \mathrm{<span\; class="notranslate">\; the\; s</span>}<span\; class="notranslate">\; .</span>\mathrm{<span\; class="notranslate">\; t</span>}<span\; class="notranslate">\; .</span>& {<span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; p</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; p</span>}}_{\mathrm{<span\; class="notranslate">\; 0</span>}\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; \&le;</span>\mathrm{<span\; class="notranslate">\; w</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; p</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; )</span>\end{array}\right.<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 5</span>}<span\; class="notranslate">\; )</span>$

Among them, it is assumed that there are N overlapping points between two adjacent projectors, x _idst represents the target position of the i-th point among them, x _iB represents the position of the i-th point after Bessel optimization, and the objective function f( p) _represents the average stitching error of all coincident points; p _j represents the current position of the j-th Bezier control point, p _0j represents the initial position of the j-th Bezier control point, and w(p _j ) represents the Moving range, constraint condition st||p _j -p _0j || ₂ ≤ w(p _j ) represents the moving range limitation of each Bezier control point.

Table 1 and Table 2 show the projection coincidence errors before and after image geometric adjustment, respectively. Figure 6 shows the actual projected image before and after correction.

Table 1 Projection coincidence error before image geometric adjustment

Table 2 Projection coincidence error after image geometric adjustment

Although the present invention is described with reference to the preferred embodiments, the above examples do not constitute a limitation of the protection scope of the present invention, and any modifications, equivalent replacements and improvements within the spirit and principles of the present invention should be included in the scope of the present invention. within the scope of the claims.

Claims (2)

1. A full-automatic geometric stitching correction method for a projection image of multiple projectors is characterized by comprising the following specific processes:

step one, arranging a camera in front of a screen and utilizing a space point X on the screen_wAnd a camera pixel point x_cCorresponding relation, calibrating pose of camera relative to screen [ R ]_c|T_c]；

Step two, establishing pixel points x of each projector_pAnd a camera pixel point x_cBased on the pose [ R ]_c|T_c]And a screenShape parameters to obtain spatial point X on screen_wAnd a projector pixel point x_pTo calibrate the pose [ R ] of each projector relative to the screen_p|T_p]；

Step three, aiming at any two projectors with overlapped projection areas, defining the projector corresponding to the right projection area in the overlapped projection area as a right projector, and defining the projector corresponding to the left projection area as a left projector; point X on the screen_wnewCorresponding pixel point x on right projector_rMapping each projector pixel point to the left projector by using the mapping relation between the projector pixel points and the camera pixel points to obtain X_wnewCorresponding pixel point x in left projector_lnew；

Step four, aiming at the left projector of any two projectors with superposed projected images, according to the corresponding point x_pAnd X_wAnd corresponding point x_lnewAnd X_wnewAnd calibrating and updating the pose [ R ] of each left projector relative to the screen for the second time_p|T_p]；

Step five, utilizing the pose [ R ] of each projector relative to the screen_p|T_p]And generating a projection image of each projector.

2. The fully-automatic geometric stitching correction method for multi-projector projection images according to claim 1, wherein after the step five is executed, the projection images are geometrically deformed by using an optimization method shown in the following formula, and the deformed correction images are used as the projection images of the projectors;

<math> <mfenced open='{' close=''> <mtable> <mtr> <mtd> <mo>m</mo> <mo>i</mo> <mo>n</mo> </mtd> <mtd> <mi>f</mi> <mrow> <mo>(</mo> <mi>p</mi> <mo>)</mo> </mrow> <mo>=</mo> <mfrac> <mn>1</mn> <mi>N</mi> </mfrac> <munderover> <mi>&Sigma;</mi> <mrow> <mi>i</mi> <mo>=</mo> <mn>1</mn> </mrow> <mi>N</mi> </munderover> <msubsup> <mrow> <mo>|</mo> <mo>|</mo> <msub> <mi>x</mi> <mi>idst</mi> </msub> <mo>-</mo> <msub> <mi>x</mi> <mi>iB</mi> </msub> <mo>|</mo> <mo>|</mo> </mrow> <mn>2</mn> <mn>2</mn> </msubsup> </mtd> </mtr> <mtr> <mtd> <mi>s</mi> <mo>.</mo> <mi>t</mi> <mo>.</mo> </mtd> <mtd> <msub> <mrow> <mo>|</mo> <mo>|</mo> <msub> <mi>p</mi> <mi>j</mi> </msub> <mo>-</mo> <msub> <mi>p</mi> <mrow> <mn>0</mn> <mi>j</mi> </mrow> </msub> <mo>|</mo> <mo>|</mo> </mrow> <mn>2</mn> </msub> <mo>&le;</mo> <mi>w</mi> <mrow> <mo>(</mo> <msub> <mi>p</mi> <mi>j</mi> </msub> <mo>)</mo> </mrow> </mtd> </mtr> </mtable> </mfenced> </math>

wherein, N coincident points, x, exist between the images projected by two adjacent projectors_idstIndicating the target position, x, of the ith coincidence point therein_iBShowing the position of the ith coincident point after Bessel optimization, an objective function f (p) showing the average splicing error of all coincident points, p_jIndicating the current position, p, of the jth Bessel control point_0jDenotes the initial position of the jth Bessel control point, w (p)_j) Represents p_jThe range of movement of (1).