CN104374374B - 3D environment dubbing system and 3D panoramas display method for drafting based on active panoramic vision - Google Patents

Abstract

The invention discloses a 3D environment replication system based on active panoramic vision, which includes omnidirectional visual sensors, a moving body laser light source, and a microprocessor for performing 3D panoramic reconstruction and 3D panoramic rendering output on omnidirectional images. The controller includes three parts: calibration, 3D reconstruction, and 3D panoramic display rendering. The 3D panoramic display rendering part mainly includes: an object-centered panoramic rendering module, a human-centered perspective display rendering module, and a panoramic perspective cycle display rendering The module, the stereo perspective view display drawing module, the panoramic stereo view cycle display drawing module and the observation distance change stereo perspective view display drawing module. The invention also discloses a 3D panoramic display drawing method based on active panoramic vision. The present invention realizes the perfect unification of geometric accuracy, realism, panoramic 3D scene rendering and display with immersive visual sense, and reconstruction process automation of 3D panoramic model reconstruction.

Description

3D environment replication system and 3D panoramic display rendering method based on active panoramic vision

technical field

The invention relates to the application of laser light source, omnidirectional visual sensor and computer vision technology in stereoscopic vision measurement and 3D rendering, in particular to a 3D environment replication system and 3D panoramic display rendering method based on active panoramic vision.

Background technique

3D reconstruction technology includes 3D measurement and 3D reconstruction. It is an emerging application technology with great development potential and practical value. The reconstruction technology of 3D model mainly involves the following three aspects: 1) The accuracy of geometry ; 2) Realism; 3) Automation of the reconstruction process. The data required for the reconstruction of the 3D model mainly includes two aspects: the depth image data of the laser scanning and the image data collected by the image sensor.

The current 3D laser scanner still has a lot of room for improvement, 1) such as precise hardware structure, which requires high-quality integration of CCD technology, laser technology, precision mechanical sensing technology, etc., resulting in expensive Manufacturing costs and maintenance costs. 2) The existing 3D laser scanning technology belongs to the surface scanning imaging technology, and a scanned point cloud image cannot obtain the whole picture of the building, especially the whole picture of the interior of the building; the point clouds obtained from different scanning stations (viewpoints) use their respective local coordinate system, so they need to be registered to a unified coordinate system. During the registration process, there are multiple transformations between multiple coordinate systems, which cause various errors and affect the calculation speed and resources. 3) More interference will be introduced in the point cloud collection process, which leads to the need to preprocess the point cloud data. 4) The point cloud data in the software configured by the 3D laser scanners of various manufacturers lacks a unified data standard, and it is difficult to achieve data sharing, which will be particularly prominent in the construction of digital cities. 5) Two different devices are used to obtain the geometric and color information of spatial object points. The quality of geometric and color information data registration between different devices directly affects the effect of texture mapping and texture synthesis. 6) The 3D modeling process requires multiple manual interventions, and the modeling efficiency is not high, which requires operators to have high professional knowledge and affects the degree of automation.

The Chinese invention patent application number is 201210137201.7, which discloses an omnidirectional three-dimensional modeling system based on an active panoramic vision sensor. The processor, the moving body laser light source completes a vertical scan to obtain the slice point cloud data at different heights, and these data are stored with the height value of the moving body laser light source as the index, so that the slice points can be The cloud data is generated sequentially and accumulated, and finally a panoramic 3D model with geometric information and color information is constructed. However, there are two main problems in this technical solution. One is that the point cloud data obtained by scanning the moving body laser light source cannot obtain plane point cloud data perpendicular to the moving body laser light source, such as the plane of the desktop, the ground, and the zenith of the house; The current 3D reconstruction software cannot satisfy the human-centered 3D rendering and display. The existing 3D software is mainly aimed at object-centered 3D rendering and display. To satisfy the human-centered 3D display, the observer must have the ability to The 3D reconstruction and reproduction of its environment, such as the 3D digitization of the Longmen Grottoes, the ultimate goal of its 3D digitization is to allow anyone to remotely visit the Longmen Grottoes through the Internet, and appreciate the artistic charm of the Longmen Grottoes from multiple angles and all-round experience. In ergonomics, visual displays and other related components are generally designed on the basis of static vision. In order to allow people to have an immersive visual sense when browsing the displayed scenes through the Internet, it is necessary to use ergonomics methods to achieve a human-centered 3D stereo rendering display.

Contents of the invention

In order to overcome the deficiencies of the existing passive panoramic stereo vision measurement devices, such as high computer resource consumption, poor real-time performance, poor practicability, and low robustness, the active three-dimensional stereo panoramic vision measurement device with full-color panoramic LED light source is vulnerable to environmental light. The present invention provides an active panorama-based system that can reduce the consumption of computer resources, quickly complete the measurement, have good real-time performance, strong practicability, and high robustness by directly acquiring the positional geometric information and color information of three-dimensional points in space. The all-round 3D modeling system of the visual sensor adopts the rendering display mode that combines the object-centered 3D macro vision and the human-centered 3D mesoscopic vision to increase the user's immersive experience.

In order to realize the content of the above invention, several core problems must be solved: (1) Realize a moving body laser light source that can cover the entire reconstructed scene; (2) Realize an active panoramic vision that can quickly obtain the depth information of the actual object sensor; (3) a method for quickly fusing laser scanning spatial data points with corresponding pixels in a panoramic image; (4) a highly automated 3D scene reconstruction method based on regular point cloud data; (5) human-centered 3D Panoramic rendering and display technology; (6) rendering and display technology that combines object-centered 3D macro vision and human-centered 3D mid-macro vision; (7) automation of the 3D reconstruction process reduces manual intervention, and the entire scanning, processing, The process of generating, drawing and displaying is completed in one go.

The technical solution adopted by the present invention to solve its technical problems is:

A 3D environment replication system based on active panoramic vision, including an omnidirectional visual sensor, a moving body laser light source, and a microprocessor for 3D panoramic reconstruction and 3D panoramic rendering output of omnidirectional images. The omnidirectional visual sensor is installed on On the guide support rod of the moving laser light source,

The moving body laser light source also includes a volume laser light source that moves up and down along the guide support rod, the volume laser source has a first omni-directional laser that is vertical to the guide support rod, and is inclined at θ _c with the axis of the guide support rod. The second omni-directional laser and the third omni-directional laser with an inclination of θ _a with the axis of the guide support rod;

The microprocessor is divided into a calibration part, a 3D reconstruction part and a 3D panoramic display drawing part;

The calibration part is used to determine the calibration parameters of the omnidirectional visual sensor, and analyze the first omnidirectional laser, the second omnidirectional laser and the third omnidirectional laser on the panoramic image taken by the omnidirectional visual sensor. Laser projection information corresponding to the laser;

The 3D reconstruction part calculates the point cloud geometric information of the moving surface according to the position of the laser light source of the moving body and the relevant pixel coordinate values of the laser projection information, and combines the point cloud geometric information of the moving surface with each omni-directional The color information of the surface laser is fused to construct a panoramic 3D model;

The 3D panoramic display drawing part includes:

The human-centered perspective display rendering module draws the human-centered perspective according to the panoramic 3D model, and the view angle and field of view of the observer in the 3D reconstruction environment.

The 3D panoramic display drawing part also includes:

The panoramic perspective circulation display drawing module draws a human-centered panoramic perspective circulation display diagram according to the panoramic 3D model, and the cyclic change and field of view of the observer in the 3D reconstruction environment;

The stereoscopic perspective display drawing module generates a right viewpoint image, a left viewpoint image and a left and right viewpoint image according to the perspective diagram to draw a human-centered stereoscopic perspective diagram;

The panoramic stereogram cyclic display drawing module, according to the panoramic 3D model, as well as the cyclic change of the view angle and the field of view range of the observer in the 3D reconstruction environment, generates left and right stereoscopic images under the azimuth angle β by constantly changing the azimuth angle β Image pair, to draw a human-centered panoramic stereoscopic perspective cycle display;

The stereoscopic perspective display drawing module for changing observation distance draws a human-centered panoramic stereoscopic view when the observation distance is constantly changing according to the panoramic 3D model, as well as the change of the observer's observation distance and field of view in the 3D reconstruction environment Perspective display diagram.

A 3D panoramic display drawing method based on the above-mentioned 3D environment replication system, comprising steps:

1) The omnidirectional visual sensor is used to shoot the panoramic image formed by the projection of the moving laser light source;

2) According to the panoramic image, determine the calibration parameters of the omnidirectional visual sensor, and analyze the laser projection information corresponding to the first omnidirectional laser, the second omnidirectional laser and the third omnidirectional laser;

3) Calculate the point cloud geometric information of the moving surface according to the position of the laser light source of the moving body and the relevant pixel coordinate values of the laser projection information, and compare the point cloud geometric information of the moving surface with the color information of each omnidirectional laser Fusion to build a panoramic 3D model;

4) Draw a human-centered perspective view according to the panoramic 3D model, and the observer's viewing angle and field of view in the 3D reconstruction environment; the specific steps are as follows:

STEP1) Set up a three-dimensional columnar space coordinate system with the single viewpoint of the omnidirectional visual sensor as the coordinate origin O _m (0,0,0);

STEP2) Determine the size of the perspective window according to the visual range of the human eye, take the azimuth β and height h as the variables of the perspective window, and obtain the point cloud data (h, β, r) corresponding to the perspective window;

STEP3) Generate a data matrix according to the step size of the azimuth angle β and the range of the height h, and the point cloud data (h, β, r);

STEP4) All three-dimensional coordinates in the data matrix are connected with triangular patches, and the color of the connecting line adopts the color average value of the two points connected;

STEP5) Output and display all the connected triangle faces to complete the human-centered perspective drawing.

The 3D panoramic display drawing method further includes drawing a human-centered panoramic perspective cycle display map, and generating a display data matrix under the azimuth angle β by continuously changing the azimuth angle β to complete the drawing of the panoramic perspective cycle display map.

The described 3D panorama display drawing method also includes drawing a human-centered stereo perspective view, and the specific drawing algorithm is as follows:

5.1: Determine the initial azimuth angle β ₁ , read the minimum distance h _min and maximum distance h _max data, determine the viewpoint, central eye=0, left viewpoint=1, right viewpoint=2;

5.2: _h =hmin;

5.3: Read the data from the initial azimuth angle β ₁ to β ₁ +300 of the distance value h, if β ₁ +300≥1000, then β ₁ +300=β ₁ +300-1000; according to the determined viewpoint, select The central eye 0 calculates the cylindrical coordinates of the spatial object point in the left and right eyes; selects the left viewpoint 1 to calculate the cylindrical coordinates of the spatial object point in the right eye, and uses the original coordinate data as the cylindrical coordinates of the left eye; selects the right viewpoint 2 to calculate the spatial object point in the left The cylindrical coordinates of the eye, the original coordinate data is used as the cylindrical coordinates of the right eye; these values are used as a new row of data of the left viewpoint matrix and the right viewpoint matrix respectively with the left and right viewpoints, h=h+Δh;

5.4: Judging h≥h _max , if not satisfied, go to 5.3 until the display matrix of the left and right viewpoints is generated;

5.5: Connect all the three-dimensional coordinates in the display matrix with triangle patches. The connection method is to first connect the data of each row with a straight line, then connect the data of each column with a straight line, and finally connect (i, The point cloud data of j) and (i+1, j+1) are connected by a straight line; the color of the connecting line adopts the color average value of the two connected points;

5.6: Through the binocular stereo display of the stereo image pair generated after the above processing, the rendering of the human-centered stereo perspective drawing is completed.

The 3D panoramic display drawing method also includes drawing a human-centered panoramic stereoscopic perspective cycle display map, by constantly changing the azimuth angle β, saving the display data matrix of the current β, respectively using triangle meshes for the left viewpoint matrix and the right viewpoint matrix The connection is made, and the resulting stereo pair is output for stereo display.

The 3D panorama display drawing method also includes drawing a human-centered stereoscopic perspective view of changes in observation distance, according to the observer constantly changing his own spatial position in the 3D environment, and determining the field of view from the perspective of ergonomics to draw the panorama stereogram.

The first omnidirectional laser, the second omnidirectional laser and the third omnidirectional laser are blue line laser, red line laser and green line laser respectively, and the blue line laser and green line laser are installed on the red line The upper and lower sides of the laser, and the axes of all line lasers intersect at a point on the axis of the guide support rod.

The described 3D panoramic display rendering method also includes object-centered panoramic rendering display, according to the point cloud data of the 3D panoramic model with the single viewpoint O _m of the omnidirectional visual sensor as the coordinate origin, scanning the panoramic scene with a moving body laser light source Scanning slices generated during scanning, extracting point cloud data generated by blue, red and green omni-directional laser projections, generating a point cloud data matrix, and realizing object-centered panoramic rendering and display.

The beneficial effects of the present invention are mainly manifested in:

(1) Provide a brand-new stereoscopic vision acquisition method, using the characteristics of omnidirectional laser scanning and omnidirectional vision to make the reconstructed 3D model have high precision and good texture information at the same time;

(2) It can effectively reduce the consumption of computer resources, and has the advantages of good real-time performance, strong practicability, high robustness, and high degree of automation, and the entire 3D reconstruction does not require manual intervention;

(3) The use of omni-directional laser detection ensures the accuracy of geometry, and the use of high-resolution panoramic image acquisition technology enables each pixel on the panoramic image to have both geometric information and color information, thereby ensuring the realism of 3D reconstruction , the whole process is automatically scanned, automatically analyzed and calculated, there is no pathological calculation problem of 3D reconstruction, and the automation of the 3D reconstruction process is realized; the geometric accuracy, realism and reconstruction process automation of the 3D panoramic model reconstruction are realized. Unite;

(4) A human-centered 3D panoramic rendering display technology is realized, which integrates object-centered 3D vision and human-centered 3D vision, so that the drawn panoramic 3D scene has a more immersive visual sense.

Description of drawings

Fig. 1 is a structural diagram of an omnidirectional vision sensor;

Figure 2 is a single-view point catadioptric omnidirectional vision sensor imaging model, Figure 2 (a) perspective imaging process, Figure 2 (b) sensor plane, Figure 2 (c) image plane;

Fig. 3 is a schematic diagram of the structure of the moving body laser light source;

Fig. 4 is a calibration explanatory diagram of an active panoramic vision sensor;

Fig. 5 is the hardware structural diagram of the all-round three-dimensional modeling system based on active panoramic vision sensor;

Fig. 6 is a structural diagram of an omnidirectional laser generator component, Fig. 6 (a) is a front view of an omnidirectional laser generator component, and figure (b) is a top view of an omnidirectional laser generator component;

Fig. 7 is the imaging principle diagram of omnidirectional vision sensor;

Figure 8 is a schematic diagram of perspective imaging;

Fig. 9 is a schematic diagram of binocular stereo imaging for changing the observation distance, Fig. 9 (a) is a schematic diagram of binocular stereo imaging before changing the observation distance, and Fig. 9 (b) is a schematic diagram of binocular stereo imaging after changing the observation distance;

Fig. 10 is a software architecture diagram of an all-round three-dimensional modeling system based on an active panoramic vision sensor and 3D panoramic drawing;

Fig. 11 is an explanatory diagram of the point cloud spatial geometric information calculation in the omni-directional three-dimensional modeling system based on the active panoramic vision sensor;

12 is a schematic diagram of a sliced panoramic image obtained when the omnidirectional three-dimensional modeling system based on the active panoramic vision sensor acquires three-dimensional point cloud data;

Fig. 13 is an explanatory diagram of the process of analyzing point cloud spatial geometric information calculation on a panoramic image;

Figure 14 is an explanatory diagram of the horizontal field of view of the human eye;

Figure 15 is an explanatory diagram of the vertical field of view of the human eye;

FIG. 16 is an explanatory diagram of acquiring green, red and blue laser projection lines respectively on the analyzed panoramic slice image.

detailed description

Example 1

Referring to Figures 1 to 16, a 3D environment replication system based on active panoramic vision and a 3D panoramic display rendering method, including omnidirectional visual sensors, moving laser light sources, and 3D panoramic reconstruction and 3D panoramic rendering of omnidirectional images output of the microprocessor.

The center of the omnidirectional vision sensor and the center of the moving body laser light source are arranged on the same axis line. As shown in Figure 1, the omnidirectional visual sensor includes a hyperboloid mirror 2, an upper cover 1, a transparent semicircular cover 3, a lower fixing seat 4, a camera unit fixing seat 5, a camera unit 6, a connecting unit 7, and an upper cover 8 . The hyperboloid mirror surface 2 is fixed on the upper cover 1, and the connecting unit 7 connects the lower fixing base 4 and the transparent semicircular cover 3 into one body, and the transparent semicircular cover 3, the upper cover 1 and the upper cover 8 are fixed together by screws, The camera unit 6 is fixed on the camera unit holder 5 with screws, and the camera unit holder 5 is fixed on the lower holder 4 with screws, and the output of the camera unit 6 in the omnidirectional visual sensor is connected with the microprocessor.

As shown in Figure 3, the mobile laser light source is used to generate a three-dimensional structure projection light source, including: guide support rod 2-1, laser generating combination unit 2-2, chassis 2-3, linear motor moving rod 2-4, Linear motor assembly 2-5, blue line laser generating unit 2-6, red line laser generating unit 2-7, green line laser generating unit 2-8.

A total of 12 holes are opened on the laser generating combination unit 2-2, and every 4 holes form a group, which are divided into the blue line laser generating unit installation hole group, the red line laser generating unit installation hole group and the green line laser generating unit Mounting hole set. The axis lines of the mounting holes of the four red line laser generating units are respectively in an orthogonal relationship with the axis lines of the cylinder of the laser generating unit 2-2, and the axis lines of the mounting holes of the four blue line laser generating units are respectively perpendicular to The axis line of the cylinder of the laser generation combination unit 2-2 is in a relationship of inclination _θc , and the axis lines of the mounting holes of the four green line laser generation units are respectively aligned with the axis line of the cylinder of the laser generation combination unit 2-2. Slope theta _a relationship. These 12 holes are evenly distributed at an angle of 90° on the circumferential direction of the cylinder of the laser generation combination unit 2-2, which ensures that the axis lines of the 12 holes intersect with the cylinder of the laser generation combination unit 2-2. On the same point on the axis line, as shown in accompanying drawing 6.

The blue line laser generating unit 2-6 is fixed in the hole of the blue line laser generating unit mounting hole group of the laser generating combination unit 2-2, as shown in accompanying drawing 6, the blue line laser after such combination can form One emits a blue omnidirectional laser light source; red line laser generating unit 2-7 is fixed in the hole of the red line laser generating unit mounting hole group of online laser generating combination unit 2-2, as shown in accompanying drawing 6, through such The combined red line laser can form a red omnidirectional laser light source; the green line laser generating unit 2-8 is fixed in the hole of the green line laser generating unit mounting hole group of the line laser generating combination unit 2-2, as attached As shown in Figure 6, the combined green line laser can form a green omnidirectional laser light source. When the 12 holes in the laser generating combination unit 2-2 are all fixed with the corresponding line laser generating units, a volume laser light source is formed; in this way, the volume laser light source can respectively project blue, red and green 3 The red omni-directional laser is in a vertical relationship with the guide support rod, the blue omni-directional laser is in a θ _c inclination angle relationship with the axis of the guide support rod, and the green omni-directional laser is in a relationship with the guide support rod The axis line is in the θ _a tilt angle relationship.

The assembly relationship of the moving body laser light source is that the body laser light source is inserted into the guide support rod 2-1 to form a moving pair, the guide support rod 2-1 is vertically fixed on the chassis 2-3, and the linear motor assembly 2-5 is fixed on the chassis 2-3, the upper end of the linear motor moving rod 2-4 is fixedly connected with the volume laser light source, and the linear motor assembly 2-5 is controlled to realize the up and down movement of the linear motor moving rod 2-4, thereby driving the volume laser light source on the guide support rod Under the guidance of 2-1, it moves up and down to form a moving volume laser light source, so that the reconstructed panorama is scanned. The linear motor assembly 2-5 is a miniature AC linear reciprocating geared motor with a reciprocating range of 700mm, model 4IK25GNCMZ15S500, a linear reciprocating speed of 15mm/s, and a maximum thrust of 625N.

The omnidirectional visual sensor is installed on the guide support rod 2-1 in the laser light source of the mobile body through the connecting plate, forming an active omnidirectional visual sensor, as shown in accompanying drawing 4, the omnidirectional visual sensor communicates with the microprocessor through the USB interface connected.

The application software of the microprocessor is mainly composed of three parts: calibration, 3D reconstruction and 3D panoramic display drawing. The calibration part mainly includes: video image reading module, omnidirectional visual sensor calibration module, omnidirectional laser information analysis module, and joint calibration module. The 3D reconstruction part mainly includes: the video image reading module, the position estimation module of the linear motor of the moving body laser light source, the omnidirectional laser information analysis module, the calculation module of the point cloud geometric information of the moving surface, the geometric information of the point cloud and A color information fusion module, a panoramic 3D model construction module based on the position information of the moving surface, a 3D panoramic model generation module and a storage unit. The 3D panoramic display and drawing part mainly includes: object-centered panoramic drawing module, human-centered perspective display and drawing module, panoramic perspective display and drawing module, stereoscopic perspective display and drawing module, panoramic stereogram display and drawing module and observation The distance-varying stereo perspective view shows the drawing module.

The video image reading module is used to read the video image of the omnidirectional visual sensor and store it in the storage unit, and its output is connected with the omnidirectional visual sensor calibration module and the omnidirectional laser information analysis module.

Omni-directional visual sensor calibration module, used to determine the parameters of the mapping relationship between three-dimensional space points and two-dimensional image points on the camera imaging plane, as shown in Figure 2; the specific calibration process is to wrap the calibration board around the omni-directional visual sensor , shoot several groups of panoramic images, establish several equations of spatial points and pixel points in the imaging plane, use the optimization algorithm to find the optimal solution, and the calculation results are shown in Table 1, which is the omnidirectional visual sensor used in the present invention calibration parameters;

Table 1 Calibration results of ODVS

After the internal and external parameters of the omnidirectional vision sensor are calibrated, the corresponding relationship between the pixels of the imaging plane and the incident light, that is, the incident angle, can be established, as shown in formula (1);

In the formula, α represents the incident angle of the point cloud, ||u″|| is the distance from a point on the imaging plane to the center point of the plane, a ₀ , a ₁ , a ₂ and a _N are the calibrated omnidirectional vision sensor For internal and external parameters, a table of correspondence between any pixel point on the imaging plane and the incident angle is established through formula (1).

After the calibration of the omnidirectional visual sensor adopted in the present invention, the point ||u″|| on the imaging plane and the incident angle α relationship of the point cloud can be represented by the following equation;

In the present invention, the two limit positions of the moving body laser light source are determined by the maximum stroke of the linear motor assembly in the moving body laser light source and the projection angle of the body laser light source. The upper limit position is set so that the eyes of an adult are in a head-up state when standing The height of the upper limit position is used as the benchmark, the initial value of the upper limit position is set to 1500mm, the lower limit position is set based on the height of the eyes of an adult when squatting down, and the initial value of the lower limit position is set to 800mm; the maximum stroke of the linear motor assembly is 700mm, there is still a 30° upward angle of view at the upper limit position, and a 30° downward angle of view at the lower limit position; the omnidirectional visual sensor adopted in the present invention has a 28° upward angle of view and a 65° downward angle of view, covering Nearly 93 ° vertical field of view and 360 ° horizontal field of view; according to the design of the present invention, the distance between the moving body laser light source and the single viewpoint 0 _m of the omnidirectional vision sensor is calculated with formula (3);

In the formula, is the distance from the ground of the single-view point O _m of the omnidirectional visual sensor, h _{up limit} is the upper limit position of the laser light source of the moving body, h _LaserMD is the moving distance of the laser light source of the moving body, h(z) is the distance between the laser light source of the moving body and the full The distance between the single viewpoint O _m of the orientation vision sensor; as shown in accompanying drawing 4.

It is stipulated here that the acquisition image rate of the omni-directional visual sensor is 15 Flame/s, and the linear reciprocating speed on the vertical direction of the moving body laser light source is set to be 15mm/s in the present invention, and the speed of the moving body laser light source between two frames is 15 mm/s. The linear movement distance in the vertical direction is 1mm, and the distance between the two extreme positions is 700mm, so the time to complete a scan in the vertical direction is 47s, and a total of 700 panoramic slice images will be generated; 700 frames need to be processed during a vertical scan In the image, there are three projection lines of the blue laser line, the red laser line and the green laser line in one frame of the image, and the images of the first frame and the 700th frame are the scanning panoramic slice images of the two extreme positions.

The all-round laser information analysis module is used to analyze the laser projection information on the panoramic image. The method of analyzing the blue laser, red laser and green laser projection points on the panorama is based on the fact that the pixel brightness of the blue laser, red laser and green laser projection points is greater than the average brightness on the imaging plane. First, the panoramic image The RGB color space is converted into the HIS color space, and then 1.2 times the average brightness on the imaging plane is used as the threshold for extracting the blue laser, green laser and red laser projection points, and the blue laser, red laser and green laser projection are extracted After pointing, it is necessary to further distinguish the blue laser, red laser and green laser projection points. In the present invention, it is judged according to the hue value H in the HIS color space. If the hue value H is between (225, 255), it is judged as a blue laser For the projection point, if the hue value H is between (0, 30), it is judged as a red laser projection point, if the hue value H is between (105, 135), it is judged as a green laser projection point, and the rest of the pixels are judged as interference ; In order to obtain the exact position of the laser projection line, the present invention adopts the Gaussian approximation method to extract the center position of the laser projection line, and the specific implementation algorithm is:

Step1: Set the initial azimuth angle β=0;

Step2: On the panoramic image, search the green laser, red laser and blue laser projection points from the center point of the panoramic image at the azimuth angle β. For the azimuth angle β, there are several consecutive pixels projected by a certain color laser, here Select the I component in the HIS color space, that is, three consecutive pixels whose brightness value is close to the highest value, and use the Gaussian approximation method to estimate the center position of the laser projection line; the specific calculation method is given by formula (4),

In the formula, f(i-1), f(i) and f(i+1) are the brightness values of three adjacent pixels close to the highest brightness value respectively, d is the correction value, and i represents the first pixel starting from the center point of the image i pixels. Therefore, the estimated central position of the color laser projection line is (i+d), which corresponds to ||u″|| in formula (1); since the order of color laser projection on the panoramic image is green laser , red laser and blue laser, so in order to exclude the influence of other projection noise on the analysis of laser information;

Step3: Change the azimuth and continue to search for the laser projection point, that is, β=β+Δβ, Δβ=0.36;

Step4: Determine the azimuth angle β=360, if it is true, the search ends; otherwise, go to Step2.

Another omnidirectional laser information analysis method is the laser projection point extraction algorithm based on the difference between frames. method, when the moving laser surface is scanning up and down, there will be obvious differences between frames in the vertical direction, that is, on different slices, and the two frames are subtracted to obtain the absolute value of the brightness difference between the two frames, and judge it Whether it is greater than the threshold to analyze and extract the laser projection points in the panoramic slice image; and then determine the green laser, red laser and blue laser projection light on a certain azimuth angle β according to the order in which the color laser projections appear on the panoramic image.

Joint calibration, used to calibrate the active omnidirectional vision sensor. Due to the inevitable existence of various assembly errors in the assembly process of the omnidirectional vision sensor and the moving laser light source, these errors can be reduced to a minimum by joint calibration. The specific method is: first, place the active omnidirectional vision sensor in a hollow cylinder with a diameter of 1000 mm, and coincide the axis line of the active omnidirectional vision sensor with the axis line in the hollow cylinder, as shown in Figure 4 As shown; then, make the moving body laser light source ON, emit red and green laser light, adjust the moving body laser light source to the upper limit position h _{up lim it} , and collect a panoramic image, observe the blue aperture and red aperture on the panoramic image Whether the center of the circle and the green aperture is consistent with the center of the panoramic image, and detect the roundness of the blue aperture, red aperture and green aperture on the panoramic image. If the center is inconsistent or the circularity does not meet the requirements, the omnidirectional vision sensor needs to be adjusted and the connection between the laser light source of the moving body; further, adjust the laser light source of the moving body to the lower limit position h _{down limit it} , and collect a panoramic image, and observe whether the centers of the blue aperture, red aperture and green aperture on the panoramic image are Consistent with the center on the panoramic image, detect the roundness of the blue aperture, red aperture and green aperture on the panoramic image. If the center is inconsistent or the roundness does not meet the requirements, it is necessary to adjust the omnidirectional vision sensor and the moving laser light source. connection between; finally, store the upper limit position h _{up lim it} , the lower limit position h _down limit it , the maximum moving distance h _LaserMD of the moving body laser light source, and the calibration parameter information of the omnidirectional vision sensor in the joint calibration database, so that Called during 3D reconstruction.

In the present invention, a high-definition imaging chip is used in the omnidirectional visual sensor, which has a resolution of 4096×2160; the moving step of the moving body laser light source is 1mm, and the vertical scanning range is 700mm, so the resolution of the sliced panoramic image produced by the moving body laser light source The ratio is 700, so that one vertical scan can complete the sampling and fusion of the geometric information and color information of each pixel on the panoramic image until the three-dimensional reconstruction and rendering display output, as shown in Figure 10.

For the 3D reconstruction part, the processing flow is:

StepA: Read the panoramic video image through the video image reading module;

StepB: Estimate the position of the linear motor of the moving body laser light source according to the moving speed of the linear motor and the time to reach the two limit points;

StepC: Analyze the omni-directional laser information on the panoramic image, and calculate the geometric information of the moving surface point cloud;

StepD: Read the panoramic video image without laser projection from the memory, and fuse the geometric information and color information of the moving surface according to the processing results in StepC;

StepE: Build a panoramic 3D model step by step;

StepF: Determine whether the limit point has been reached, if yes, go to StepG, if not, go to StepA;

StepG: Set the moving body laser light source to OFF, read the panoramic video image without laser projection, and save it in the memory unit, output the 3D panoramic model and save it to the storage unit, set the moving body laser light source to ON, turn to Step A.

The following is a detailed description of the processing flow of 3D reconstruction. In StepA, a thread is specially used to read the panoramic video image. The reading rate of the video image is 15 Flame/s. The collected panoramic image is stored in a memory unit, so that Subsequent processing calls;

In StepB, it is mainly used to estimate the current position of the moving body laser source; it is stipulated that the initial position of the moving body laser source is set at the upper limit position h _{up lim it} at the beginning of reconstruction, and the initial step length control value z _move (j) = 0, the moving step of the moving body laser light source in two adjacent frames is Δz, that is, the following relationship exists,

z _move (j+1)=z _move (j)+Δz (5)

In the formula, z _move (j) is the time step control value of the jth frame, z _move (j+1) is the time step control value of the j+1th frame, and Δz is the moving step size of the laser light source of the moving body. Here, When moving downward from the upper limit position h _{up lim it} , Δz=1mm; when moving upward from the lower limit position h _{down lim it} , Δz=-1mm; when the program is implemented, it is judged by the following relational formula,

Substituting the calculation result value z _move (j+1) of formula (5) into h _LaserMD in formula (3), the distance h(z) between the laser light source of the moving body and the single-view point O _m of the omnidirectional vision sensor is obtained.

In StepC, the panoramic image in the memory unit is read and the omnidirectional laser information analysis module is used to analyze the omnidirectional laser information from the panoramic image, and then the geometric information of the moving surface point cloud is calculated.

If the spatial position information of the point cloud is represented by the Gaussian coordinate system, the spatial coordinates of each point cloud relative to the single-view point O _m of the omni-directional visual sensor are determined by three values for the Gaussian coordinate origin of the Gaussian coordinate origin, namely (R, α, β), R is the distance from a certain point cloud to the single-viewpoint O _m of the omnidirectional vision sensor, α is the incident angle from a certain point cloud to the single-viewpoint O _m of the omnidirectional vision sensor, and β is a certain point cloud To the azimuth of the single viewpoint O _m of the omnidirectional vision sensor, for the point cloud in Figure 13 with point, the calculation method of point cloud data in Gaussian coordinates is given by formulas (7), (8), and (9),

In the formula, (β) _green is the azimuth angle of the green laser projection point cloud to the single-view point O _m of the omnidirectional vision sensor, (β) _red is the azimuth angle of the red laser projection point cloud to the single-view point O _m of the omnidirectional vision sensor , (β) _blue is the azimuth angle of the blue laser projection point cloud to the single-viewpoint O _m of the omnidirectional vision sensor, θ _B is the angle between the blue projection line and the Z axis, θ _G is the green projection line and the Z axis The angle between the axes, h(z) is the distance from the moving body laser light source to the single-view point O _m of the omnidirectional vision sensor, α _a is the incident angle of the green laser projection point cloud to the single-view point O _m of the omnidirectional vision sensor , α _b is the incident angle of the red laser projection point cloud to the single-viewpoint O _m of the omnidirectional vision sensor, α _c is the incident angle of the blue laser projection point cloud to the single-viewpoint O _m of the omnidirectional vision sensor, R _a is green The distance from the laser projection point cloud to the single-viewpoint _Om of the omnidirectional vision sensor, _Rb is the distance from the red laser projection point cloud to the single-viewpoint _Om of the omnidirectional vision sensor, and _Rc is the distance from the blue laser projection point cloud to the omnidirectional The distance of the single-viewpoint O _m of the visual sensor, ||u"||(β) _green is the distance between the corresponding point of the green laser projection point on the imaging plane and the center of the panoramic imaging plane, ||u"||(β ) _red is the distance between the corresponding point of the red laser projection point on the imaging plane and the center of the panoramic imaging plane, ||u"||(β) _blue is the corresponding point of the blue laser projection point on the imaging plane to the panoramic imaging The distance between the centers of the planes.

If the point cloud with point in Cartesian coordinate system with To express, with reference to accompanying drawing 11, its calculation method is given by formula (10), (11), (12),

In the formula, R _a is the distance from the green laser projection point cloud to the single-viewpoint _Om of the omnidirectional vision sensor, _Rb is the distance from the red laser projection point cloud to the single-viewpoint _Om of the omnidirectional vision sensor, and _Rc is blue The distance from the laser projection point cloud to the single-viewpoint _Om of the omnidirectional vision sensor, α _a is the incident angle of the green laser projection point cloud to the single-viewpoint _Om of the omnidirectional vision sensor, _αb is the distance from the red laser projection point cloud to the omnidirectional The incident angle of the single-view point O _m of the visual sensor, α _c is the incident angle of the blue laser projected point cloud to the single-view point O _m of the omnidirectional visual sensor, (β) _green is the incident angle of the green laser projected point cloud to the omnidirectional visual sensor The azimuth of the single-view point O _m , (β) _red is the azimuth of the single-view point O m from the red laser projection point cloud to the omnidirectional vision sensor, (β) _blue is the single-view point O _m from the blue laser projection point cloud to the omnidirectional vision sensor The azimuth of the viewpoint O _m .

In the StepC calculation process, the point cloud data generated by the omnidirectional 360° blue, red and green omnidirectional laser projections have been traversed; since the high-definition imaging chip is used in the present invention, in order to achieve consistency with the vertical scanning accuracy, the calculation method is used here The step size is Δβ=0.36 to traverse the entire 360° azimuth angle. Attached figure 16 is the panorama of the scanning result of the moving body laser light source at a certain height position. The green short dotted line on the panorama is produced by the green omnidirectional laser projection point cloud data The red long dotted line is the point cloud data generated by the red omnidirectional laser projection The blue short dotted line is the point cloud data generated by the blue omnidirectional laser projection The traversal algorithm is described in detail below;

StepⅠ: Set the initial azimuth angle β=0;

Step Ⅱ: Use the omnidirectional laser information analysis module to retrieve the point cloud along the ray direction with Get the three points ||u″||(β) _green , ||u″||(β) _red and ||u"||(β) _blue corresponding to the point cloud data on the imaging plane, using the formula (7) Calculate point cloud The distance value R _a and the incident angle α _a , use the formula (8) to calculate the point cloud The distance value R _b and the incident angle α _b , use the formula (9) to calculate the point cloud The distance value R _c and the incident angle α _c ; then use the formula (10) to calculate the point cloud in the Cartesian coordinate system Calculate the point cloud with formula (11) in the Cartesian coordinate system Calculate the point cloud with formula (12) in the Cartesian coordinate system In this calculation step, the ergodic azimuth β is respectively substituted into (β) _green , (β) _red , (β) _blue in the formulas (10), (11), and (12); the above calculation data are saved in in the memory unit;

StepⅢ: β←β+Δβ, Δβ=0.36, judge whether β=360 is true, if it is true, end the calculation, otherwise go to StepⅡ.

In StepD, the panoramic video image without laser projection is first read from the memory, and the geometric information and color information of the point cloud are fused according to the processing results in StepC; the fused point cloud data will include the geometry of the point cloud Information and color information, that is, use (R, α, β, r, g, b) to express the geometric information and color information of a certain point cloud. The fusion algorithm will be described in detail below.

Step①: Set the initial azimuth angle β=0;

Step②: According to the azimuth β and the information of the two points ||u″||(β) _red and ||u″||(β) _green corresponding to the point cloud data on the sensor plane, read the non-laser projection In this case, the (r, g, b) color data of the relevant pixels on the panoramic video image are fused with the corresponding (R, α, β) obtained from the processing in StepC to obtain the corresponding point cloud geometry information and color information (R,α,β,r,g,b);

Step③: β←β+Δβ, Δβ=0.36, judge whether β=360 is true, if true, end the calculation, save the calculation result in the storage unit; otherwise, go to Step②.

In StepE, the panoramic 3D model is gradually built according to the calculation results of StepD. In the present invention, the moving body laser light source completes a vertical scanning process, that is, the construction of the panoramic 3D model is completed from one extreme position to another extreme position. Each moving step in the scanning process will generate slice point cloud data at a certain height, as shown in Figure 12; these data are stored with the height value of the laser light source of the moving body as the index, so that the slice point cloud data can be The cloud data generation sequence is accumulated to finally build a panoramic 3D model with geometric information and color information; according to the above description, the present invention has two different modes of downward panoramic 3D reconstruction and upward panoramic 3D reconstruction;

In StepF, it is judged whether the laser light source of the moving body has reached the limit position, that is, it is judged whether z _move (j)=0 or z _move (j)=h _LaserMD is established, and if it is established, it is turned to StepG, and if it is not established, it is turned to StepA.

In StepG, the main job is to output the reconstruction results and make some preparations for the next reconstruction; the specific method is: first set the moving body laser light source to OFF, read the panoramic video image without laser projection, and save it In the memory unit; then output the 3D reconstructed panorama model and save it to the storage unit, because in the present invention, no matter whether it is in the generation of slice point cloud data or in the aspect of omnidirectional point cloud data generation on a certain slice, high resolution is adopted. Each pixel on the imaging plane has the geometric information and color information corresponding to the actual point cloud, so it effectively avoids the problems of corresponding points, tiles and problems in 3D reconstruction. Branch problem; finally set the moving body laser light source to ON, go to Step A, and reconstruct the new 3D panoramic model.

Through the above processing, the point cloud data of the 3D panoramic model with the single-view point O _m of the omnidirectional visual sensor as the coordinate origin is obtained; when scanning the panoramic scene with the laser projection light source of the panoramic mobile surface, scanning slices are generated one by one, as shown in Figure 12 As shown; the blue part in the figure is the point cloud data generated by the blue omnidirectional laser projection, the red part is the point cloud data generated by the red omnidirectional laser projection, and the green part is the green omnidirectional laser projection Project the resulting point cloud data.

These scanning slice images are naturally formed with the movement of the laser projection light source on the panoramic moving surface during the scanning process. After each scan obtains a panoramic slice image, the azimuth angle is used to traverse the entire panoramic slice image to extract blue, red and green full-scale images. The point cloud data generated by laser projection on the azimuth plane; the point cloud data is stored in a matrix, in which the 1-700 rows of the matrix store the point cloud data generated by the blue omni-directional laser projection, and the 701-1400 rows of the matrix store the red The point cloud data generated by omnidirectional laser projection, the 1401~2100 rows of the matrix store the point cloud data generated by green omnidirectional laser projection, the number of columns indicates the number of azimuth scans from 0° to 359.64°, a total of 1000 columns ; Therefore, the point cloud storage matrix is a matrix of 2100×1000; each point cloud includes (x, y, z, R, G, B) 6 attributes, which forms an ordered point cloud data set, with The advantage of ordinal data set is that its neighborhood operation can be more efficient after knowing the relationship between adjacent points in advance.

The object-centered panorama rendering module is used to read the data in the point cloud storage matrix and then call a pcl_visualization library in PCL, through which the obtained point cloud data files can be quickly 3D modeled and visualized. Realize object-centered panorama rendering and display.

The human-centered perspective view display drawing module is used to draw a 3D perspective view according to the view angle and field of view of the observer in the 3D reconstruction environment. The drawing algorithm steps are as follows:

STEP1) Take the monoscopic point of the panoramic vision sensor as the origin of coordinates O _m (0,0,0), and establish a three-dimensional cylindrical space coordinate system;

STEP 2) Determine the size of the perspective window, based on the visual range of the human eye, about 108° in the width direction, and use the azimuth angle β as a variable; use h as a variable in the height direction;

STEP 3) When obtaining the detection results of the active panoramic vision sensor, the point cloud data (h, β, r) with the single viewpoint of the panoramic vision sensor as the coordinate origin O _m (0, 0, 0) is obtained, and the height direction The range is determined by the minimum distance h _min and the maximum distance h _max , and there are N rows of data in total; considering that in the width direction, the azimuth β is continuously obtained with a step of 0.36° angle, and each slice scan In the process, 1000 point cloud data will be generated; here, the left edge of the perspective window is taken as the initial azimuth β ₁ , then the right edge of the perspective window is β ₁ +300, and there are M columns of data, where M=300;

STEP 4) Determine the first data according to the selected initial azimuth angle β ₁ , such as the initial azimuth angle β ₁ = 36°, then select the 100th data and start to 400 data as the first column at the minimum distance h _min Data, starting from 1100 data to 1400 data is the second column data of h _min +Δh,...

STEP 5) Processing of the display data matrix of the perspective window, such as the determination of the minimum distance h _min and maximum distance h _max obtained so far, with a total of 2100 rows of data; the specific algorithm is as follows:

STEP51: Determine the initial azimuth angle β ₁ , read the minimum distance h _min and maximum distance h _max data;

STEP52: h=h _min ;

STEP53: Read the data from the initial azimuth angle β ₁ to β ₁ +300 at the distance h, if β ₁ +300≥1000, then β ₁ +300=β ₁ +300-1000; use these values as a new row of the matrix Data, h=h+Δh;

STEP54: Judging h≥h _max , if not satisfied, go to STEP53;

STEP55: end.

Obtain a 80×300 matrix through the above algorithm;

STEP 6) Connect all the three-dimensional coordinates in the matrix with triangle patches. The connection method is to first connect the data of each row with a straight line, then connect the data of each column with a straight line, and finally connect the (i , j) and (i+1, j+1) point cloud data are connected by a straight line; the color of the connecting line is the average color of the two connected points;

STEP 7) Display all connected triangles on the output device.

The working principle of the omnidirectional vision sensor is: the light entering the center of the hyperboloid mirror is refracted toward its virtual focus according to the specular characteristics of the hyperboloid. The object image is reflected by the hyperboloid mirror into the condenser lens for imaging, and a point P (x, y) on the imaging plane corresponds to the coordinate A (X, Y, Z) of a point of the object in space.

In accompanying drawing 7, 2—hyperbolic mirror, 12—incident light, 13—real focus Om(0,0,c) of hyperboloid mirror, 14—virtual focus of hyperboloid mirror, that is, center Oc of camera unit 6 (0,0,-c), 15-reflected light, 16-imaging plane, 17-space coordinates A(X,Y,Z) of the real image, 18-space coordinates of the image incident on the hyperboloid mirror surface, 19 - Point P(x,y) reflected on the imaging plane.

The optical system that the hyperboloid mirror shown in accompanying drawing 7 constitutes can be expressed by following 5 equations;

((X ² +Y ² )/a ² )-((Zc) ² /b ² )=-1 when Z>0 (20)

β=tan ^-1 (Y/X) (22)

α=tan ^-1 [(b ² +c ² )sinγ-2bc]/(b ² +c ² )cosγ (23)

In the formula, X, Y, and Z represent the space coordinates, c represents the focal point of the hyperbolic mirror, 2c represents the distance between the two focal points, a, b are the lengths of the real axis and imaginary axis of the hyperbolic mirror, respectively, and β represents the incident light The angle between the X-axis and the X-axis on the XY projection plane is the azimuth angle. α represents the angle between the incident light and the X-axis on the XZ projection plane. Here, α is called the incident angle, and when α is greater than or equal to 0, it is called the depression angle. When α is less than 0, it is called the elevation angle, f represents the distance from the imaging plane to the virtual focus of the hyperbolic mirror, γ represents the angle between the refraction light and the Z axis; x, y represent a point on the imaging plane.

Example 2

The other structure and working process of this embodiment are the same as those of Embodiment 1, the difference is: to meet the needs of different reconstruction scenarios, replace the hardware module of the moving body laser light source, that is, change the projection angle of the green laser line in Figure 4 θ _a and the projection angle θ _c of the blue laser line to change the vertical scanning range.

Example 3

The rest are the same as in Embodiment 1. According to the requirements of drawing and displaying different 3D scenes, the human-centered panoramic perspective view cycle display drawing module is used to continuously change the viewing angle in the 3D reconstruction environment according to the observer. From the perspective of ergonomics Determine the field of view to draw a 3D perspective, the core of which is to continuously and slowly change the azimuth angle β to generate a display data matrix under the azimuth angle β; the key algorithm is as follows:

STEP1: Determine the initial azimuth angle β ₁ , read the minimum distance h _min and maximum distance h _max data;

STEP2: β=β ₁ , determine the cycle display times N, n=0;

STEP3: h=h _min ;

STEP4: Read the data from the initial azimuth β of the distance value h to β+300, if β+300≥1000, then β+300=β+300-1000; use these values as a new row of data in the matrix, h= h+Δh;

STEP5: Judging h≥h _max , if not satisfied, go to STEP4;

STEP6: Save the display data matrix in the current β, connect the display data matrix with triangle patches, and display the output; β=β+Δβ;

STEP7: Judging that β≥3600;

STEP8: If β=β-3600, n=n+1;

STEP9: Judging n≥N, if not satisfied, go to STEP3;

STEP10: end.

Example 4

The other structure and working process of this embodiment are the same as those of Embodiment 1, the difference is: for different 3D scene rendering and display requirements, a human-centered stereo perspective view display rendering module is used to continuously change the 3D weight according to the observer. According to the angle of view in the structural environment, the scope of the field of view is determined from the perspective of ergonomics to draw a 3D stereoscopic perspective; that is, when the perspective generated by the human-centered perspective display drawing module is used as a left viewpoint image, a right viewpoint is generated Image; in case of anthropocentric perspective displaying the generated perspective of the rendering module as a right viewpoint image, a left viewpoint image is generated; in anthropocentric perspective displaying the generated perspective of the rendering module as a central eye image In this case, left and right viewpoint images are generated to realize stereo pair generation; therefore, the geometric relationship between the single-viewpoint coordinate O _m (0,0,0) of ODVS and the spatial object point P(h,β,r) is required Calculate the coordinates of the new viewpoint;

Using the single-viewpoint coordinates O _m (0,0,0) of ODVS as the coordinates of the right-eye viewpoint, use the formula (13) to calculate the coordinates of the point P in the left-eye viewpoint;

In the formula, h _R is the height of point P in space at the viewpoint of the left eye, h _L is the height of point P in space at the viewpoint of the right eye, r _R is the incident angle of point P in space at the viewpoint of the left eye, is the square value of the incident angle of point P in space at the viewpoint of the right eye, β _L is the azimuth of point P in space at the viewpoint of the left eye, and β _R is the azimuth of point P in space at the viewpoint of the right eye;

Using the single-viewpoint coordinates O _m (0,0,0) of ODVS as the coordinates of the left-eye viewpoint, use formula (14) to calculate the coordinates of the space P point at the right-eye viewpoint;

In the formula, h _R is the height of point P in space at the viewpoint of the left eye, h _L is the height of point P in space at the viewpoint of the right eye, r _R is the incident angle of point P in space at the viewpoint of the left eye, is the square value of the incident angle of point P in space at the viewpoint of the right eye, β _L is the azimuth of point P in space at the viewpoint of the left eye, and β _R is the azimuth of point P in space at the viewpoint of the right eye;

Using the single-viewpoint coordinates _Om (0,0,0) of ODVS as the coordinates of the central eye, use formula (15) to calculate the coordinates of the point P in the left and right eyes;

In the formula, h _R is the height of point P in space at the viewpoint of the left eye, h _L is the height of point P in space at the viewpoint of the right eye, r _R is the incident angle of point P in space at the viewpoint of the left eye, is the square value of the incident angle of point P in space at the viewpoint of the right eye, β _L is the azimuth of point P in space at the viewpoint of the left eye, and β _R is the azimuth of point P in space at the viewpoint of the right eye;

B is the distance between the eyes. The distance between the eyes of women is 56-64mm, and the distance between the eyes of men is 60-70mm. Here, a distance of 60mm that is acceptable to both sexes is taken, that is, B=60;

The stereo perspective display algorithm is as follows:

STEP1: Determine the initial azimuth angle β ₁ , read the minimum distance h _min and maximum distance h _max data, determine the viewpoint, central eye=0, left viewpoint=1, right viewpoint=2;

STEP2: h=h _min ;

STEP3: Read the data from the initial azimuth angle β ₁ to β ₁ +300 of the distance value h, if β ₁ +300≥1000, then β ₁ +300=β ₁ +300-1000; according to the determined viewpoint, if Select 0 (central eye) and use formula (15) to calculate the cylindrical coordinates of the spatial object point in the left and right eyes; if you select 1 (left viewpoint), use formula (14) to calculate the cylindrical coordinates of the spatial object point in the right eye, and use the original coordinate data as Cylindrical coordinates of the left eye; if 2 (right viewpoint) is selected, use formula (13) to calculate the cylindrical coordinates of the space object point in the left eye, and use the original coordinate data as the cylindrical coordinates of the right eye; use the left and right viewpoints respectively and use these values as the left viewpoint Matrix and a new row of data of the right view matrix, h=h+Δh;

STEP4: Judging h≥h _max , if not satisfied, go to STEP3;

STEP5: end;

Through the above algorithm, two 80×300 matrices are obtained, which are the display matrices of the left and right viewpoints respectively.

Connect all the three-dimensional coordinates in the matrix with triangle patches. The connection method is to first connect the data of each row with a straight line, then connect the data of each column with a straight line, and finally connect the (i, j ) and (i+1, j+1) point cloud data are connected by a straight line; the color of the connecting line is the average color of the two connected points.

The stereoscopic image pair generated after the above processing is then subjected to binocular stereoscopic display.

For binocular stereo display supported by the graphics card. For example, in the OpenGL environment that supports stereoscopic display, start the stereoscopic display mode of OpenGL in the stage of creating a device handle, and send the generated stereoscopic image pairs to the left and right buffers to realize stereoscopic display.

For graphics cards that do not support stereoscopic display, the stereoscopic image pair is synthesized into a red-green complementary color stereoscopic image, and the red channel is extracted from one image in the left and right stereoscopic image pair, and the green and blue channels are extracted from the other image. The channels are fused to form a complementary color stereoscopic image.

Example 5

The other structure and working process of this embodiment are the same as those of Embodiment 1, the difference is: for different 3D scene drawing and displaying requirements, the human-centered panoramic stereogram display drawing module is used to constantly change the 3D weight according to the viewer. The angle of view in the structural environment is determined from the perspective of ergonomics to draw a panoramic stereogram. The core is to continuously change the azimuth β to generate left and right stereo pairs under the azimuth β; the key algorithm is as follows:

STEP1: Determine the initial azimuth angle β ₁ , read the minimum distance h _min and maximum distance h _max data, determine the viewpoint, central eye=0, left viewpoint=1, right viewpoint=2;

STEP2: β=β ₁ , determine the cycle display times N, n=0;

STEP3: h=h _min ;

STEP4: Read the data from the initial azimuth angle β ₁ to β ₁ +300 of the distance value h, if β ₁ +300≥1000, then β ₁ +300=β ₁ +300-1000; according to the determined viewpoint, if Select 0 (central eye) and use formula (3) to calculate the cylindrical coordinates of the spatial object point in the left and right eyes; if you select 1 (left viewpoint), use formula (2) to calculate the cylindrical coordinates of the spatial object point in the right eye, and use the original coordinate data as Cylindrical coordinates of the left eye; if you choose 2 (right viewpoint), use formula (1) to calculate the cylindrical coordinates of the space object point in the left eye, and use the original coordinate data as the cylindrical coordinates of the right eye; use the left and right viewpoints respectively and use these values as the left viewpoint Matrix and a new row of data of the right view matrix, h=h+Δh;

STEP5: Judging h≥h _max , if not satisfied, go to STEP4;

STEP6: Save the display data matrix in the current β, respectively connect the left viewpoint matrix and the right viewpoint matrix with triangular patches, and generate stereoscopic image pairs for stereoscopic display output; β=β+Δβ;

STEP7: Judging that β≥3600;

STEP8: If β=β-3600, n=n+1;

STEP9: Judging n≥N, if not satisfied, go to STEP3;

STEP10: end.

Example 6

The other structure and working process of this embodiment are the same as those of Embodiment 1, the difference is: for different 3D scene rendering and display requirements, the human-centered observation distance change stereoscopic perspective display rendering technology is used to continuously Change your own spatial position in the 3D environment, determine the field of view from the perspective of ergonomics to draw a panoramic stereogram; when the observer roams the scene from far to near or from near to far, the distance between the observer and the reconstructed space The position of the point cloud has changed, which involves the change of point cloud coordinates and multi-level display technology, and the amount of point cloud data used for display will also change accordingly.

First, let’s observe how the spatial position of the point cloud changes when the observer roams the scene from far to near; when considering the spatial position of the point cloud, the calculation method of the central eye is still used here, that is, the midpoint between the two eyes is used as the observer's point of view;

When the observer roams the scene from far to near, the spatial P(h,β,r) becomes P(h _D ,β _D ,r _D ) relative to the observer's point of view. Since it is cumbersome to perform the above calculations on the cylindrical coordinate system, we use the direction of observation as the Y axis, so when the observer roams the scene from far to near, he only moves a distance D on the Y axis. Therefore, we use the point cloud P(h, β, r) of the cylindrical coordinate system first to use the formula (16);

Point cloud P(x,y,z) converted into Cartesian coordinate system; According to the calculation results, in order not to lose generality, the new viewpoint O' _m (0,0,0) and the original viewpoint O _m (0,0 ,0) is D, which is expressed by formula (17) and formula (18),

Use the formula (19) to calculate the coordinates P(x',y',z') of all point clouds in the new viewpoint O' _m (0,0,0) coordinate system, and then convert the points of the Cartesian coordinate system The point cloud P(x',y',z') is converted into the point cloud P(h',β',r') of the Gaussian coordinate system. For the case of approaching the roaming scene in the Y-axis direction, a simple calculation can be performed, that is, y'=y+y ₀ . Take the moved viewpoint O' _m (0,0,0) as the coordinates of the central eye, and use the formula (19) to calculate the coordinates of the spatial point P(h',β',r') in the left and right eye viewpoints; for moving In the case of the later viewpoint O' _m (0,0,0), compared with the original viewpoint O _m (0,0,0), the field of view will change accordingly, and the horizontal field of view of the field of view specified above is 108 °, the vertical field of view is 66°; therefore, the field of view at the new viewpoint O' _m (0,0,0) needs to be calculated by the new viewpoint O' _m (0,0,0) on the calculation result of formula (19) The incident angle α' and the azimuth angle β' are determined, and then the stereoscopic perspective at the new viewpoint O' _m (0,0,0) is drawn using the ASODVS-based anthropocentric stereoscopic rendering technique; for the new The human-centered panoramic stereoscopic cyclic display rendering of the viewpoint O' _m (0,0,0) uses the human-centric panoramic stereoscopic cyclic display rendering technology based on ASODVS;

In the formula, h' _L is the height of point P in the right eye viewpoint in the new viewpoint O' _m (0,0,0) coordinate system, and r' _L is the new viewpoint O' _m (0,0,0) In the coordinate system, the incident angle of the space point P at the right eye viewpoint, β' _L is the azimuth angle of the space P point at the left eye viewpoint under the new viewpoint O' _m (0,0,0) coordinate system, and h' _R is the new The viewpoint O' _m (0,0,0) coordinate system of the space P point is at the height of the left eye viewpoint, and r' _R is the new viewpoint O' _m (0,0,0) coordinate system, and the space point P is on the left The angle of incidence of the eye point of view, β' _R is the azimuth angle of point P in the right eye point of view in the new point of view O' _m (0,0,0) coordinate system.

When displaying a new viewpoint, first judge the distance from the new viewpoint to the root node. When the distance is relatively close, that is, when the point cloud data used for display is less than a certain threshold, the interpolation from the adjacent root node data is further generated. Hierarchical point cloud data; generate low-level point cloud data, that is, interpolation operation is a basic operation in volume rendering. Due to its large amount of calculation, it is realized through the fast interpolation algorithm in volume rendering.

Claims (9)

1. a kind of 3D environment dubbing systems based on active panoramic vision, including omnibearing vision sensor, moving body laser light Source and the microprocessor for omnidirectional images to be carried out with the reconstruct of 3D panoramas and the drafting output of 3D panoramas, all-directional vision sensing Device is arranged on the guiding support bar of moving body LASER Light Source, it is characterised in that：

Described moving body LASER Light Source also includes the volumetric laser light source moved up and down along guiding support bar, volumetric laser light source tool There are the first comprehensive face laser of vertically-guided support bar and the axial line of guiding support bar into θ_cThe full side of angle inclined second Plane laser and with the axial line of guiding support bar into θ_aThe comprehensive face laser of angle the inclined 3rd；

Described microprocessor is divided into demarcation part, 3D reconstruct part and the display of 3D panoramas and draws part；

Described demarcation part, the calibrating parameters for determining omnibearing vision sensor, and in omnibearing vision sensor institute The first comprehensive face laser, the second comprehensive face laser and the 3rd comprehensive face laser correspondence are parsed on the panoramic picture of shooting Laser projection information；

Described 3D reconstruct part, according to the position of moving body LASER Light Source, and the laser projection information related pixel Coordinate value, calculates the point cloud geological information in mobile face, and by the point cloud geological information in mobile face and the face of each comprehensive face laser Color information is merged, and builds panorama 3D models；

Part is drawn in described 3D panoramas display, including：

Perspective view focusing on people shows drafting module, is reconstructed in 3D according to described panorama 3D models, and observer Perspective view focusing on people is drawn at visual angle and field range in environment.

2. the 3D environment dubbing systems of active panoramic vision are based on as claimed in claim 1, it is characterised in that described 3D is complete Part is drawn in scape display, is also included：

Full-view perspective circulation display drafting module, is in 3D and reconstructs environment according to described panorama 3D models, and observer The varying cyclically and field range at middle visual angle draw panoramic perspective focusing on people circulation display figure；

Perspective view shows drafting module, according to the perspective view focusing on people, generates right visual point image, left view point Image and left and right visual point image draw perspective view focusing on people；

Full-view stereo figure circulation display drafting module, is in 3D and reconstructs environment according to described panorama 3D models, and observer The varying cyclically and field range at middle visual angle, by constantly changing azimuthal angle beta, generate the left and right space image under the azimuthal angle beta It is right to draw full-view stereo focusing on people perspective circulation display figure；

With viewing distance change perspective view display drafting module, according to described panorama 3D models, and observer is in 3D Panorama focusing on people when the change of viewing distance and field range constantly change viewing distance to be plotted in reconstruct environment Volume rendering display figure.

3. a kind of 3D panoramas based on 3D environment dubbing system described in claim 1 or 2 show method for drafting, it is characterised in that Including step：

1) panoramic picture that moving body laser light source projects are formed is shot using omnibearing vision sensor；

2) panoramic picture according to, determines the calibrating parameters of omnibearing vision sensor, and parse the first comprehensive face Laser, the second comprehensive face laser and the corresponding laser projection information of the 3rd comprehensive face laser；

3) according to the position of moving body LASER Light Source, and the laser projection information related pixel coordinate value, calculate movement The point cloud geological information in face, and the colouring information of the point cloud geological information in mobile face and each comprehensive face laser is merged, Build panorama 3D models；

4) visual angle and field range that the panorama 3D models according to, and observer are in 3D reconstruct environment are drawn Perspective view focusing on people；Comprise the following steps that：

STEP1 it is) origin of coordinates O with the single view of omnibearing vision sensor_m(0,0,0), sets up three-dimensional column space coordinates System；

STEP2) according to the visual range of human eye, the size of see-through window is determined, with azimuthal angle beta and height h as see-through window Variable, and (h, β, r), r is distance of the correspondence spatial point to single view to obtain the corresponding cloud data of see-through window；

STEP3 step-length and the scope of height h) according to azimuthal angle beta, and described cloud data (h, β r), generate data square Battle array；

STEP4) all of three-dimensional coordinate in data matrix is coupled together with triangle surface, the color of connecting line is using connection Two color averages of point；

STEP5 the triangle surface of all connections) is carried out into output display, perspective view focusing on people is completed and is drawn.

4. 3D panoramas as claimed in claim 3 show method for drafting, it is characterised in that described 3D panoramas display method for drafting Also include drawing panoramic perspective circulation display figure focusing on people, by constantly changing azimuthal angle beta, generate in the azimuthal angle beta Under display data matrix, complete panoramic perspective circulation display figure draw；The algorithm of the display data matrix is as follows：

4.1) initial orientation angle beta is determined₁, read minimum range h_minWith ultimate range h_maxData；

4.2) β is entered as β₁, it is determined that circulation display number of times is N, n initial values are 0；

4.3) h is entered as h_min；

4.4) the initial orientation angle beta of reading distance value h starts the data to β+300, if β+300 >=1000, the assignment of β+300 It is β+300-1000；Using these β value as matrix new data line, h is entered as h+ Δs h；

4.5) h >=h is judged_maxIf being unsatisfactory for going to 4.4)；

4.6) the display data matrix of current β is stored in, display data matrix is attached with triangle surface, shown defeated Go out；β is entered as β+Δ β；

4.7) β >=3600 are judged；

4.8) if it is, β is entered as β -3600, n is entered as n+1；

4.9) n >=N is judged, if being unsatisfactory for going to 4.3).

5. 3D panoramas as claimed in claim 3 show method for drafting, it is characterised in that described 3D panoramas display method for drafting Also include drawing perspective view focusing on people, specific rendering algorithm is as follows：

5.1：Determine initial orientation angle beta₁, read minimum range h_minWith ultimate range h_maxData, determine viewpoint；

5.2：H is entered as h_min；

5.3：Read the initial orientation angle beta of distance value h₁Start to β₁+ 300 data, if β₁+ 300 >=1000, then β₁+ 300 assign It is β to be worth₁+300-1000；According to identified viewpoint, selection median eye calculates cylindrical coordinates of the dimensional target point in right and left eyes；Selection Left view point calculate dimensional target point right eye cylindrical coordinates, using former coordinate data as left eye cylindrical coordinates；Right viewpoint is selected to calculate Dimensional target point left eye cylindrical coordinates, using former coordinate data as right eye cylindrical coordinates；These values are made with left and right viewpoint respectively It is left view dot matrix and the new data line of right viewpoint matrix, h is entered as h+ Δs h；

5.4：Judge h >=h_maxIf being unsatisfactory for going to 5.3, until the display matrix of generation left and right viewpoint；

5.5：All of three-dimensional coordinate in display matrix is coupled together with triangle surface respectively, connection method is first will be every The data of a line are attached with straight line, and then the data of each row are attached with straight line, finally by (i, j) in matrix and The cloud data of (i+1, j+1) is attached with straight line；The color of connecting line is using two color averages of point for connecting；

5.6：The stereogram generated after being processed to more than carries out binocular solid and shows, completes focusing on people three-dimensional saturating View Drawing.

6. 3D panoramas as claimed in claim 3 show method for drafting, it is characterised in that described 3D panoramas display method for drafting Also include drawing full-view stereo perspective circulation display figure focusing on people, by constantly changing azimuthal angle beta, generate in the orientation Left and right stereogram under angle beta, its algorithm is as follows：

6.1：Determine initial orientation angle beta₁, read minimum range h_minWith ultimate range h_maxData, on median eye, left view point and the right side Determine viewpoint among viewpoint three；

6.2：β is entered as β₁, it is determined that circulation display times N, n initial values are 0；

6.3：H is entered as h_min；

6.4：Read the initial orientation angle beta of distance value h₁Start to β₁+ 300 data, if β₁+ 300 >=1000, then β₁+ 300 assign It is β to be worth₁+300-1000；According to identified viewpoint, selection median eye calculates cylindrical coordinates of the dimensional target point in right and left eyes；Selection Left view point calculate dimensional target point right eye cylindrical coordinates, using former coordinate data as left eye cylindrical coordinates；Right viewpoint is selected to calculate Dimensional target point left eye cylindrical coordinates, using former coordinate data as right eye cylindrical coordinates；These values are made with left and right viewpoint respectively It is left view dot matrix and the new data line of right viewpoint matrix, h is entered as h+ Δs h；

6.5：Judge h >=h_maxIf being unsatisfactory for going to 6.4；

6.6：The display data matrix of current β is preserved, left view dot matrix and right viewpoint matrix are carried out with triangle surface respectively Connection, generates stereogram, stereoscopic display output；β is entered as β+Δ β；

6.7：Judge β >=3600；

6.8：If β >=3600, β is entered as β -3600, and n is entered as n+1；

6.9：N >=N is judged, if being unsatisfactory for going to 6.3.

7. the 3D panoramas as described in claim 5 or 6 show method for drafting, it is characterised in that according to haplopia point coordinates O_m(0,0, 0) and dimensional target point P (h, β, r) between geometrical relationship, the coordinate for calculating new viewpoint includes：

By haplopia point coordinates O_m(0,0,0) calculates space P points in left eye viewpoint as the coordinate of right eye viewpoint with formula (13) Coordinate；

$\left\{\begin{array}{c}{h}_R=h_L\\ r_R=\sqrt{B^2+r_L^2+2\×B\×r_L\×{\mathrm{cos\β}}_L}\\ {\mathrm{\β}}_R=arcsin(\frac{r_R}{r_L}\×{\mathrm{sin\β}}_L)\end{array}\right.---\left(13\right)$

In formula, h_RIt is space P points in the height of left eye viewpoint, h_LIt is space P points in the height of right eye viewpoint, r_RFor P points in space exist The incidence angle of left eye viewpoint,It is space P points in the square value of the incidence angle of right eye viewpoint, β_LIt is space P points in left eye viewpoint Azimuth, β_RIt is space P points at the azimuth of right eye viewpoint；

By haplopia point coordinates O_m(0,0,0) calculates space P points in right eye viewpoint as the coordinate of left eye viewpoint with formula (14) Coordinate；

$\left\{\begin{array}{c}{h}_L=h_R\\ r_L=\sqrt{B^2+r_R^2+2\×B\×r_R\×{\mathrm{cos\β}}_R}\\ {\mathrm{\β}}_L=arcsin(\frac{r_L}{r_R}\×{\mathrm{sin\β}}_R)\end{array}\right.---\left(14\right)$

By haplopia point coordinates O_m(0,0,0) calculates space P points in right and left eyes viewpoint as the coordinate of median eye with formula (15) Coordinate；

$\left\{\begin{array}{c}{h}_L=h\\ r_L=\sqrt{{(B/2)}^2+r^2-B\×r\×cos\mathrm{\β}}\\ {\mathrm{\β}}_L=\arcsin (\frac{r_L}{r}\×sin\mathrm{\β})\\ h_R=h\\ r_R=\sqrt{{(B/2)}^2+r^2+B\×r\×cos\mathrm{\β}}\\ {\mathrm{\β}}_R=\arcsin (\frac{r_R}{r}\×sin\mathrm{\β})\end{array}\right.---\left(15\right)$

Wherein, the distance between B is two.

8. 3D panoramas as claimed in claim 3 display method for drafting, it is characterised in that described the first comprehensive face laser, Second comprehensive face laser and the 3rd comprehensive face laser are respectively blue line laser, red line laser and green line laser, blue Colo(u)r streak laser and green line laser are separately mounted to the both sides up and down of red line laser, and the axis of all line lasers intersects at institute State a bit on the axial line of guiding support bar.

9. 3D panoramas as claimed in claim 8 show method for drafting, it is characterised in that described 3D panoramas display method for drafting Also include that the panorama centered on thing draws display, according to the single view O of omnibearing vision sensor_mIt is the 3D of the origin of coordinates The cloud data of panorama model, the scan slice produced when scanning panoramic scene with moving body LASER Light Source, extracts blue, red Cloud data produced by color and green comprehensive face laser projection, generates cloud data matrix, realizes complete centered on thing Scape draws display.