JP5580164B2 - Optical information processing apparatus, optical information processing method, optical information processing system, and optical information processing program - Google Patents

Description

  The present invention relates to a technique using three-dimensional point cloud position data, and more particularly to a technique for efficiently performing processing for improving the accuracy of three-dimensional point cloud position data.

  A technique for generating a three-dimensional shape from three-dimensional point cloud position data of a measurement object is known. In the three-dimensional point cloud position data, a two-dimensional image and a three-dimensional coordinate are linked. That is, in the three-dimensional point cloud position data, the data of the two-dimensional image of the measurement object, a plurality of measurement points (point cloud) associated with the two-dimensional image, and the three-dimensional space of the plurality of measurement points in the three-dimensional space. Are associated with each other (three-dimensional coordinates). According to the three-dimensional point cloud position data, it is possible to obtain a three-dimensional model that reproduces the outer shape of the measurement object using a set of points. In addition, since the three-dimensional coordinates of each point are known, the relative positional relationship of each point in the three-dimensional space can be grasped, and the image of the three-dimensional model displayed on the screen can be rotated or switched to an image viewed from a different viewpoint. Processing is possible.

  When considering the acquisition of 3D point cloud position data from a single viewpoint, it is not possible to acquire 3D point cloud position data that is a shadow when viewed from that viewpoint due to the shape of the measurement object or an obstacle. . This phenomenon is called occlusion, and the shadowed part is called the occlusion part or the occlusion part. As described above, one of the purposes of acquiring the point cloud position data is to acquire a three-dimensional model, and the three-dimensional model is rotated so as to be in a state viewed from a desired viewpoint like a three-dimensional CAD. There is a feature in what can be done. Therefore, if there is the above-mentioned occlusion, when the acquired three-dimensional model is rotated, the three-dimensional model lacking the occlusion portion is displayed. Alternatively, there are cases where it is desired to make a drawing from the point cloud data or the three-dimensional model. However, if there is an occlusion part, there is a problem that the part cannot be made a drawing.

  In order to solve this occlusion problem, three-dimensional point cloud position data is obtained again from the second viewpoint that can obtain the three-dimensional point cloud position data of the portion that becomes occlusion from the first viewpoint. A three-dimensional model based on the three-dimensional point cloud position data may be created. However, since it takes time to acquire the three-dimensional point cloud position data, this method is not efficient.

  As an approach for solving this problem, Patent Document 1 discloses an occlusion separately from the acquisition of the basic 3D point cloud position data in order to complementarily acquire the 3D point cloud position data of the occlusion portion. Describes a method that complements the 3D point cloud position data of the occlusion part by taking a picture from another viewpoint that can photograph the part and obtaining the relationship between this captured image and the above 3D point cloud position data. ing.

JP 2008-82707 A

  The method described in Patent Document 1 is efficient because complementary data is obtained by photographing in addition to acquisition of basic three-dimensional point cloud position data. By the way, when acquiring three-dimensional point cloud position data based on a photographed image, a method based on the stereo method using a stereo pair image is used. Since the stereo method acquires the three-dimensional information of the object to be imaged based on the principle of triangulation, internal parameters such as the focal length, distortion, and principal point position of the imaging device (camera) (these parameters are called internal orientation elements) are used. If the correction taking into account is not performed, the error of the obtained three-dimensional point group position data (the error of the coordinates of each measurement point) increases.

  The process of obtaining parameters necessary for this correction is called calibration. In general, calibration of an imaging apparatus is performed when shipping as a product. However, this pre-calibration is performed under certain specific conditions (setting conditions in the factory), and the distance to the measurement object is fixed or limited in order to perform accurate correction. For this reason, in general, the shooting for acquiring the three-dimensional point cloud position data is limited in terms of conditions and is not convenient. Moreover, although it is possible to calibrate the photographing apparatus in a situation where photographing is actually performed, complicated operations such as setting of targets and setting of various conditions are required, which is not efficient.

  Against this background, the present invention provides a technique for obtaining three-dimensional point cloud position data that complements three-dimensional point cloud position data obtained using a laser scanner from a photographed image. The purpose is to provide a technology that enables efficient and simple operation.

The invention of claim 1 includes a first three-dimensional point cloud position data receiving unit that receives a three-dimensional point group location data according to the measurement object that is based on reflected light resulting laser beam in the first aspect, A coordinate calculation unit that calculates second 3D point cloud position data related to the measurement object based on a stereo image of the measurement object photographed at the second viewpoint; and the first 3D point cloud position data; based on a corresponding relationship with specific parts, said identified in the first three-dimensional point cloud position data the correspondence relationship specifying unit said correspondence relationship and identifying a correspondence between the second three-dimensional point group location data, and an internal orientation parameter calculation unit for calculating the internal orientation parameters of the imaging apparatus that has performed the shooting of the stereo image, based on the internal orientation parameters calculated in the internal orientation element calculation unit, and an image correcting section for correcting the stereo images The above 1 3D point cloud position data and 3D 3D point cloud position data related to the measurement object based on the stereo image corrected by the image correction unit, and the integrated 3D point cloud An optical information processing apparatus comprising: a three-dimensional model creating unit that creates a three-dimensional model of the measurement object based on position data .

The principle of the present invention will be briefly described. First, to obtain the three-dimensional point group location data of the measurement object by using a laser over light from the first viewpoint. Next, photographing is performed by an image photographing unit (for example, a digital camera) from a second viewpoint different from the first viewpoint. This second viewpoint is selected when selecting a viewpoint where an occlusion part can be seen from the first viewpoint, or by complementing an inaccurate part of the point cloud position data obtained at the first viewpoint. There are cases where a viewpoint suitable for obtaining is selected.

  Then, orientation is performed between the three-dimensional point cloud position data obtained at the first viewpoint and the captured image obtained at the second viewpoint, and the correspondence between the two is obtained. Since the 3D point cloud position data has 3D coordinate data of each measurement point, the 3D coordinates of the feature points in the captured image obtained from the second viewpoint can be specified by obtaining the above correspondence. It becomes possible. Then, using the feature point in the captured image obtained from the second viewpoint as a reference (target), calibration of the image capturing unit (for example, a digital camera) used for capturing from the second viewpoint is performed. Thereby, the internal orientation element of the image photographing unit at the second viewpoint is acquired. Here, the internal orientation elements of the image capturing unit are internal parameters of the image capturing unit such as lens distortion, focal length, and lens principal point position. The internal orientation element includes information such as lens system distortion. By obtaining an internal orientation element, a correction coefficient or function for eliminating or reducing image distortion can be obtained.

  According to this technique, the image capturing unit is calibrated in a situation where actual capturing is performed based on already obtained three-dimensional point cloud position data. At this time, it is only necessary to take an image as a work, and a complicated work such as placing a calibration-specific target and specifying its coordinates is not required. Then, based on the internal orientation element of the image capturing unit obtained by the above calibration, the image captured by the image capturing unit at the second viewpoint is corrected and the distortion is removed. By doing so, the accuracy of the three-dimensional point cloud position data of the measurement object based on the image obtained by photographing from the second viewpoint is improved.

  Then, 3D point cloud position data is obtained based on the corrected photographed image from the second viewpoint, and is used as 3D point cloud position data that complements the 3D point cloud position data obtained from the first viewpoint. Use. As a result, (1) elimination of occlusion, (2) complementation of the inaccurate portion of the 3D point cloud position data obtained from the first viewpoint, (3) obtained from the first viewpoint due to the influence of the passing vehicle, etc. Completion of partial missing of 3D point cloud position data, (4) Compensation of poor accuracy and missing of 3D point cloud position data obtained from the first viewpoint due to some other factor, and the like can be performed.

The invention of claim 2 is the invention according to claim 1, wherein the correspondence relationship specification unit, based on the shape of the measurement object, and performs calculation of the correspondence relation.

The invention according to claim 3 is the invention according to claim 1 or 2, wherein the correspondence specifying unit is one or more selected from the intensity, color, hue, and saturation of the reflected light of the laser beam. When, and performing certain of the relationship on the basis of the image of the measurement object.

  According to a fourth aspect of the present invention, in the first or second aspect of the invention, the correspondence specifying unit captures an image obtained by photographing the region irradiated with the laser light, and the image photographing unit captures the image. The correspondence relationship is specified based on a comparison with an image.

The invention according to claim 5 is the invention according to any one of claims 1 to 4, wherein the coordinate calculation unit is a three-dimensional point again based on the stereo image corrected by the image correction unit. By calculating group position data, the third three-dimensional point group position data is calculated, and the correspondence specifying unit is configured to calculate the first three-dimensional point group position data and the third three-dimensional point. The correspondence relationship with the group position data is specified, and the first three-dimensional point group is based on the correspondence relationship between the first three-dimensional point group position data and the third three-dimensional point group position data. The position data and the third three-dimensional point group position data are integrated .

According to a sixth aspect of the present invention, in the invention according to any one of the first to fourth aspects, the stereo image is obtained by a digital camera .

  The invention according to claim 7 is the invention according to any one of claims 1 to 6, wherein the image correction unit is the focal length information of the image photographing unit obtained in advance or the internal part in addition to the focal length information. The image capturing unit is configured to correct an image captured based on the internal orientation element calculated by the orientation element calculating unit. Here, the focal length information includes the relationship between the focal length and the internal orientation element obtained in advance. Although this focal length information includes an error, it can be used as a rough standard for the value of the internal orientation element calculated by the internal orientation element calculation unit. For this reason, it is possible to obtain effects such as reduction of errors in processing necessary for image correction, shortening of processing time, and reduction of the probability that an error occurs in processing and the computation is delayed or stopped.

  The following two forms can be mentioned as forms using the focal length information. One of them is a case where focal length information is used to correct distortion of an image captured by an image capturing unit necessary for calculating an internal orientation element. In this case, since the distortion of the image captured by the image capturing unit used for calculating the internal orientation element is reduced, the internal orientation calculation accuracy is increased.

  Another mode in which the focal length information is used is a case where the focal length information is used for calculation in the internal orientation element calculation unit. In this case, since a rough guide for the approximate value of the internal orientation element to be calculated is provided, effects such as improvement in calculation accuracy, reduction in calculation time, and reduction in the probability of occurrence of calculation errors can be obtained.

The invention of claim 8 includes a first three-dimensional point cloud position data receiving means for receiving three-dimensional point group location data according to the measurement object that is based on reflected light resulting laser beam in the first aspect, Coordinate calculation means for calculating second 3D point cloud position data relating to the measurement object based on a stereo image of the measurement object photographed at the second viewpoint, and the first 3D point cloud position data; Based on the correspondence relationship specifying means for specifying the correspondence relationship with the second 3D point group position data , the correspondence relationship specified by the first 3D point group position data and the correspondence relationship specifying means , and internal orientation parameters calculating means for calculating the internal orientation parameters of the imaging apparatus that has performed the shooting of the stereo image, based on the internal orientation parameters calculated in the internal orientation element calculation means, image Osamu modifying the stereo images Integrating means, and a third three-dimensional point group location data relating to the said object of measurement based on the stereo images have been fixed in the first three-dimensional point cloud position data said image correction means, is the integrated An optical information processing system comprising: a three-dimensional model creating means for creating a three-dimensional model of the measurement object based on the three-dimensional point cloud position data .

The invention of claim 9 includes a first three-dimensional point group location data reception step of receiving three-dimensional point group location data according to the measurement object that is based on reflected light resulting laser beam in the first aspect, A coordinate calculation step of calculating second 3D point cloud position data relating to the measurement object based on a stereo image of the measurement object photographed at the second viewpoint; and the first 3D point cloud position data; Based on the correspondence relationship identifying step for identifying the correspondence relationship with the second 3D point cloud position data, and the correspondence relationship identified in the first 3D point cloud position data and the correspondence relationship identifying step , and internal orientation parameters calculation step of calculating an internal orientation parameters of the imaging apparatus that has performed the shooting of the stereo image, based on the internal orientation parameters calculated in the internal orientation element calculation step, the stearate An image correction step of correcting the O image, a third three-dimensional point group location data relating to the said object of measurement based on the stereo images have been fixed in the first of the image correction step and the three-dimensional point group location data integrated, an optical information processing method characterized by comprising a three-dimensional model creation step of creating a three-dimensional model of the object to be measured based on the integrated three-dimensional point group location data.

The invention of claim 10 is a program to be executed is read by the computer, the first three-dimensional point group according to the measurement object that is based on the reflected light of the laser light obtained in the first aspect of the computer 3D point cloud position data receiving means for receiving position data, and coordinate calculation for calculating second 3D point cloud position data relating to the measurement object based on a stereo image of the measurement object taken at the second viewpoint Means, correspondence specifying means for specifying the correspondence between the first three-dimensional point cloud position data and the second three-dimensional point cloud position data , the first three-dimensional point cloud position data and the correspondence based on the above correspondence relationship identified in relationship specifying unit, and an internal orientation parameters calculating means for calculating the internal orientation parameters of the imaging apparatus that has performed the shooting of the stereo images, the internal orientation element calculation Based on the internal orientation parameters calculated in step, the image correction means for correcting the stereo image, the measurement object based on the first three-dimensional point cloud position data said stereo images corrected by the image correction means a third three-dimensional point group location data integrated according to, to function as a three-dimensional model creation means for creating a three-dimensional model of the object to be measured based on the integrated three-dimensional point group location data was This is an optical information processing program.

  According to the present invention, using the 3D coordinate data of the 3D point cloud position data obtained by the laser scanner, the imaging unit is calibrated under conditions for imaging. For this reason, in the technique for obtaining 3D point cloud position data that complements the 3D point cloud position data obtained by using a laser scanner from a photographed image, the photographing means is calibrated efficiently and simply under the conditions for photographing. be able to.

It is a block diagram of the optical information processing apparatus of an embodiment. It is a block diagram of a point cloud position data processing unit of an embodiment. It is explanatory drawing explaining the principle of a stereo method. It is explanatory drawing explaining a stereo image. It is explanatory drawing explaining a pass point. It is explanatory drawing explaining a relative orientation. It is explanatory drawing explaining the principle of a stereo matching. It is explanatory drawing which shows the principle of the matching using a label. It is a flowchart which shows an example of a process. It is a conceptual diagram which shows the state of a measurement. It is sectional drawing of a laser scanner. It is sectional drawing of a laser scanner. It is a block diagram of a laser scanner. It is a block diagram of a calculating part. It is a conceptual diagram which shows the three-dimensional point cloud position data whose distance between points is not constant. It is a conceptual diagram which shows the formed grid.

1. First Embodiment Hereinafter, an example of an optical information processing apparatus using the invention will be described with reference to the drawings. In the following description, the three-dimensional point cloud position data includes three-dimensional coordinate data at each measurement point of the measurement object. An orthogonal coordinate system or a polar coordinate system is adopted as a coordinate system for displaying three-dimensional coordinates. Image data refers to image data obtained by photographing with a CCD or the like.

  An optical information processing apparatus 100 is shown in FIG. The optical information processing apparatus 100 is configured as software on a personal computer. A program that configures the optical information processing apparatus 100 on a personal computer is installed in the personal computer. This program may be recorded on a server or an appropriate recording medium and provided from there.

  The personal computer used is an input unit such as a keyboard or a touch panel display, an image display device such as a liquid crystal display, a GUI (graphical user interface) function unit that is a user interface integrating the input unit and the display unit, a CPU, and the like. Dedicated arithmetic device, semiconductor memory, hard disk storage unit, disk storage device drive unit capable of exchanging information with storage media such as optical disc, and interface capable of exchanging information with portable storage media such as USB memory And a communication interface unit for performing wireless communication and wired communication as necessary. The personal computer may be a notebook type, a portable type, a desktop type, or the like, but the form is not limited. In addition to using a general-purpose personal computer, optical information processing is performed by dedicated hardware configured using an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device) such as an FPGA (Field Programmable Gate Array), or the like. It is also possible to configure the device 100.

  In the optical information processing apparatus 100, a laser scanner 121 used for acquiring three-dimensional point cloud position data of a measurement object and a digital camera 122 for photographing the measurement object and obtaining an image thereof are of appropriate standards. Connected using a communication line. The laser scanner 121 irradiates the measurement object with laser light and receives the reflected light, thereby acquiring information on the distance, azimuth, elevation angle or depression angle from the installation position of the laser scanner 121 to the measurement point. Based on the information, information related to the three-dimensional position coordinates of the measurement point is calculated. Further, the laser scanner 121 acquires information on the intensity, color, hue, saturation, and the like of reflected light from the measurement point. Based on this information, the laser scanner 121 calculates point cloud position data including the three-dimensional coordinate values of each measurement point, and outputs the data to the optical information processing apparatus 100. An example of a detailed configuration of the laser scanner 121 will be described later. Further, the laser scanner 121 includes an imaging device (CCD imaging device) having an optical axis in the same direction as the optical axis of the laser beam, and the image data of the measurement target that is the acquisition target of the three-dimensional point cloud position data. It has a structure that can be acquired.

  The digital camera 122 is an example of an image capturing unit, and captures an overlapping portion of the measurement object from different viewpoints on the left and right to acquire a stereo pair image. The stereo pair image data (one set of left and right image data) is output to the optical information processing apparatus 100. The digital camera 122 sets two digital cameras on a dedicated base and shoots a stereo pair image, and uses one digital camera to obtain a stereo pair image from each of two viewpoints. It is possible to adopt a form in which one set of left and right image data is obtained by taking a picture.

  The optical information processing apparatus 100 includes a three-dimensional point cloud position data receiving unit 101 and an image receiving unit 102. The three-dimensional point group position data receiving unit 101 receives the three-dimensional point group position data output from the laser scanner 121 and takes it into the optical information processing apparatus 100. The image receiving unit 102 receives image data of a stereo pair image taken by the digital camera 122 and takes it into the optical information processing apparatus 100.

  The optical information processing apparatus 100 includes a position specifying unit 103, a coordinate calculating unit 104, a correspondence specifying unit 105, an internal orientation element calculating unit 106, an image correcting unit 107, and a three-dimensional point cloud position data processing unit 108. Hereinafter, each of these functional units will be described.

(Position specifying unit 103)
The position specifying unit 103 specifies the three-dimensional position of the measurement object (three-dimensional coordinates of the characteristic points that characterize the appearance of the measurement object) based on the three-dimensional point cloud position data acquired by the laser scanner 121. As a method of grasping the three-dimensional position of the measurement object, a method of using the acquired three-dimensional point cloud position data as it is, or a method of using label data calculated by the three-dimensional point cloud position data processing unit 108 to be described later And a method using a three-dimensional model (three-dimensional contour model of a measurement object) calculated by a three-dimensional point cloud position data processing unit. Since the label data and the three-dimensional model data have a smaller amount of data than the basic three-dimensional point cloud position data, the burden on the CPU and memory during software processing is small, and the calculation speed can be increased.

(Coordinate calculation unit)
The coordinate calculation unit 104 uses the stereo method to calculate the three-dimensional position of the measurement object (photographing object) (the three-dimensional coordinates of the feature points that characterize the appearance of the measurement object) based on the stereo pair image obtained from the digital camera 122. To calculate. This three-dimensional position data is point group position data similar to the three-dimensional point group position data obtained by the laser scanner 121.

The contents of the calculation performed by the coordinate calculation unit 104 will be described. First, the principle of calculating the three-dimensional position of the measurement object based on the stereo pair image will be described. Hereinafter, this method is referred to as a stereo method. FIG. 3 is an explanatory diagram for explaining the principle of the stereo method. As shown in FIG. 3, the two cameras have parallel optical axes, the distances a from the principal points of the camera lenses 1 and 2 to the CCD surface as the photographing surface are equal, and the CCDs are placed at right angles to the optical axis. It shall be. Also, the distance between two optical axes (base length line) equal to the interval between the camera lenses 1 and 2 of each camera is assumed to be l. The principle is the same even if the camera is a camera for stereo photography, in which one camera is provided with camera lenses 1 and 2. At this time, between the coordinates of the point P (x, y, z) on the object and the point P ₁ (x1, y1) on the image 1 and the point P ₂ (x2, y2) on the image 2, There is a relationship like

  However, the origin of the entire coordinate system is assumed to be the principal point of the camera lens 1. Z is obtained from Equation 3, and x and y are obtained from Equation 1 and Equation 2 using this equation. As described above, if the corresponding points of the left and right images are obtained, the three-dimensional coordinates of the position can be calculated.

  Next, the principle of creating a stereo image will be described. The principle of creating a stereo image is to correct the image so that the stereo method is established. A stereo image is one in which two images are parallel to an object and vertical parallax is removed. FIG. 4 is an explanatory diagram for explaining a stereo image. For example, in the image shown in FIG. 4A, the scales of the left and right images are different, and there is rotation and vertical parallax, regardless of how much the left and right images overlap. Dimension measurement is not possible. However, if the left and right image magnifications are adjusted as shown in FIG. 4B, the rotation is corrected, and the vertical parallax is removed, the stereo method can be realized.

  A stereo image (stereo model) can be created by obtaining six or more identical corresponding points in the left and right images. The same corresponding point in the left and right images is called a pass point. FIG. 5 is an explanatory diagram for explaining pass points. As shown in FIG. 5, if there are image coordinates of points corresponding to 6 points on the left and right images, the relative position and inclination of the two cameras can be obtained. It is possible to correct the vertical parallax and create a stereo image.

  Mutual orientation is used to create the stereo image. The relative orientation is a principle for obtaining the relative position and inclination of the camera from six or more corresponding points in the image. FIG. 6 is an explanatory diagram for explaining relative orientation. In relative orientation, each parameter is obtained by the following coplanar conditional expression.

As shown in FIG. 6, the origin of the model coordinate system left nitrilase projection center O _1, to take a line connecting the right projection center O ₂ on the X axis. For the scale, the base line length is taken as the unit length. The parameters to be obtained at this time are the rotation angle κ ₁ of the left camera, the rotation angle φ _{1 of} the Y axis, the rotation angle κ _{2 of the} right camera, the rotation angle φ ₂ of the Y axis, and the rotation of the X axis. the five of the rotation angle of the corner ω _2. In this case, since the rotation angle ω ₁ of the X axis of the left camera is 0, there is no need to consider it. Under such conditions, the coplanar conditional expression of Expression 4 becomes as shown in Expression 5, and each parameter can be obtained by solving this expression.

  Here, the following relational expression for coordinate transformation is established between the model coordinate system XYZ and the camera coordinate system xyz.

Using these equations, unknown parameters are obtained by the following procedure.
(1) The initial approximate value is normally 0.
(2) The coplanar conditional expression of Expression 5 is Taylor-expanded around the approximate value, and the values of the differential coefficients when linearized are obtained by Expressions 6 and 7, and an observation equation is established.
(3) A least square method is applied to obtain a correction amount for the approximate value.
(4) The approximate value is corrected.
(5) Using the corrected approximate value, the operations (2) to (5) are repeated until convergence.

  As described above, the relative three-dimensional position of the camera and the inclination of the three axes are obtained, and a stereo image (stereo model) can be created. Moreover, if a distance (baseline length) between cameras that have taken a stereo image is input, an actual stereo model can be obtained. Furthermore, if three XYZ coordinates can be given out of the six points, the coordinates can be converted into an actual coordinate system, and three-dimensional measurement in actual coordinates can be performed. Further, if the base line length is set to 1, three-dimensional coordinate calculation in the model coordinate system is possible.

  The calculation based on the principle described above is performed in the coordinate calculation unit 104, and the three-dimensional coordinates of the object to be imaged based on the stereo pair image captured by the digital camera 122 are calculated. This calculation is performed at each grid point obtained by dividing the imaging screen showing the measurement object into a grid. Thereby, the three-dimensional coordinate distribution on the surface of the measurement object is calculated in a point cloud shape. The coordinate data of the surface of the measurement object in the form of a point group becomes the three-dimensional point group position data of the measurement object calculated based on the stereo pair image captured by the digital camera 122.

(Corresponding relationship specifying unit 105)
Based on the output of the laser scanner 121, the correspondence specifying unit 105 relates to the data related to the three-dimensional position of the measurement object specified by the position specifying unit 103 and the three-dimensional position of the measurement object calculated by the coordinate calculation unit 104. Identify correspondence between data. Hereinafter, an example of a method for specifying the correspondence between two pieces of three-dimensional data in the correspondence specifying unit 105 will be described. First, corresponding points are designated in a portion where the visual field of the measurement object from the viewpoint where the laser scanner 121 is arranged and the viewpoint where the digital camera 122 is arranged (the portion where the measurement target portion is visible from the data acquisition position) overlaps. Next, various orientations are performed based on the corresponding points, and a correspondence relationship between coordinate systems (orthogonal coordinate system or polar coordinate system) describing each of the three-dimensional point group position data obtained from the two viewpoints is calculated. Thereby, the correspondence between the first three-dimensional point group position data and the second three-dimensional point group data is specified. Hereinafter, the orientation used when calculating the correspondence between the two three-dimensional point cloud position data will be described.

(Mutual orientation)
The relative orientation is described above. In this case, the viewpoints of the left and right cameras are the first viewpoint and the second viewpoint, and the coordinate system (first coordinate system) of the three-dimensional point group position data obtained from the first viewpoint is (X ₁ , Y ₁ , Z ₁ ), and the coordinate system (second coordinate system) of the three-dimensional point cloud position data obtained from the second viewpoint as (X ₂ , Y ₂ , Z ₂ ), the first coordinate system and the second coordinate system An operation for obtaining a correspondence relationship with the coordinate system is performed.

(Single photo orientation)
Single photo orientation refers to the camera position (X0, Y0, Z0) and camera tilt (ω, φ, This is a technique for obtaining a relationship between photographic coordinates x, y, z and ground coordinates X, Y, Z. The collinear condition is a condition that the projection center, the photographic image, and the ground object are in a straight line. The camera position (X0, Y0, Z0) and the camera tilt (ω, φ, κ) are called external orientation elements.

  Here, using the principle of single photo orientation, the first coordinate system when the measurement object is viewed from the first viewpoint, and the second coordinate system when the measurement object is viewed from the second viewpoint. A method for obtaining the relationship with the will be described. In this case, in the state where the three-dimensional point cloud position data of the measurement object is acquired by the laser scanner 121 at the first viewpoint, the digital camera 122 takes a picture of the measurement object from the second viewpoint, Furthermore, the following calculation is performed in the correspondence specifying unit 105 in a state where the three-dimensional point cloud position data of the measurement object is acquired by the function of the coordinate calculation unit 104 based on this photographic image.

  First, the first coordinate system is a reference point coordinate (X, Y, Z), and the second coordinate system is a photographic coordinate (x, y, z). Specify a point as a corresponding point. These four points are common coordinate points that serve as a basis for determining the correspondence between the two coordinate systems, and a portion that is a feature point from the measurement object (for example, a point that can be easily distinguished from the surroundings by the appearance of an edge or the like). Selected. The four points can be selected by a manual operation, a method for automatically extracting a part that can be easily grasped as a feature point such as an edge or a corner of an object, or a method for automatically selecting the four points. Further, a method in which the user manually selects is used. Note that the designation of corresponding points when single photo orientation is used may be five or more, but the correspondence between the two coordinate systems can be obtained by designating at least four points.

  Then, the four screen coordinate values and the corresponding three-dimensional coordinates of the reference point are substituted into the quadratic projective transformation equation shown in the following equation 8, and an observation equation is established to obtain the parameters b1 to b8. Here, the screen coordinate values of the four points are the positions of the coordinates on the screen of the corresponding points of the four points designated from the captured image captured by the digital camera 122. The corresponding three-dimensional coordinates of the reference point are the values of the three-dimensional coordinates of the four corresponding points obtained from the three-dimensional point group position data acquired by the laser scanner 121.

  The external orientation elements (X0, Y0, Z0) are obtained from the following equation 9 using parameters b1 to b8 of equation 8.

  Next, the coordinate system (xp, yp, zp) of the tilted digital camera 122 corresponding to (X, Y, Z) is obtained from the following equation 10 based on the principle of single photo orientation. In Equation 10, the tilt (ω, φ, κ) of the digital camera 122 obtained in Equation 9 is substituted, and the rotation matrix is calculated to obtain the parameters a11 to a33.

The obtained parameters a11 to a33 and the position (X0, Y0, Z0) and (X, Y, Z) obtained in Expression 9 are expressed by the following Expression 11 in which the projection center, the photographic image, and the object are in a straight line. Substituting into the collinear conditional expression, the coordinates (x, y) are obtained. Here, c is a screen distance (focal length), a _{11 to} a ₃₃ are inclinations of the digital camera 122 expressed as elements of a 3 × 3 rotation matrix, and Δx and Δy are calculated by the internal orientation element calculation unit 106. This is a correction term based on the calculated internal orientation element in the digital camera.

  Thus, a coordinate system (first coordinate system) (X, Y, Z) when the measurement object is viewed from the first viewpoint and a coordinate system (first coordinate system) when the measurement object is viewed from the second viewpoint. 2) (x, y, z) is calculated. In the above calculation method, the relationship between z and Z is not obtained. However, when the first viewpoint and the second viewpoint have a difference in position in the horizontal plane, z = Z. No problem arises.

(Absolute orientation)
Absolute orientation is a method of converting a model coordinate system into a ground coordinate system (absolute coordinate system). When absolute orientation is used, the first coordinate system is associated with the ground coordinate system, while the second coordinate system is associated with the ground coordinate system, and the first coordinate system and the second coordinate system are connected via the ground coordinate system. Is obtained. First, the model coordinate system (XM, YM, ZM) is converted to the ground coordinate system (X, Y, Z). Here, when the scale is s, the rotation about three axes is ω, φ, κ, and the parallel movement amount is (X0, Y0, Z0), the relational expression of Formula 12 is obtained.

  Next, assuming that ω and φ are small, unknown variables (s, ω, φ, κ, X0, Y0, Z0) are obtained. First, the plane coordinates are adjusted by Helmat transform. When limited to plane coordinates, the following equation 10 holds. In Equation 13, cosκ = (a / s) and sinκ = (− b / s).

  In Equation 13, coefficients a, b, X0, and Y0 are determined by the least square method. Next, the scale is unified. In this case, the following equation 14 holds.

  Next, the height is adjusted. In this case, the following formula 15 holds.

  In Equation 15, ω, φ, and Z0 are obtained by the method of least squares. Then, using the obtained unknown variable, the model coordinates are corrected by the following equation (16).

  The above processing is repeated until the unknown variable converges, and the correspondence relationship between the model coordinate system (XM, YM, ZM) and the ground coordinate system (X, Y, Z) is obtained. Then, by selecting the first coordinate system from the first viewpoint and the second coordinate system from the second viewpoint as the model coordinate system, the first coordinate system and the second coordinate system via the ground coordinate system are selected. The correspondence with the coordinate system becomes clear. Or the image from two viewpoints and the three-dimensional point cloud position data from two viewpoints can be handled by the ground coordinate system which is a common coordinate. When absolute orientation is used, theoretically, orientation is performed by specifying three corresponding points. For example, as in the case where a coordinate system from two different viewpoints at the same height position is used as a problem. Since the position on one axis is the same even if the coordinate system is changed, the number of corresponding points required for orientation may be two.

(How to specify corresponding points)
In the above-described orientation, it is necessary to designate corresponding points that serve as a basis for calculating the correspondence between coordinate systems. As a method for designating corresponding points, there are a manual designation method, a calculation method, and a combination of the two. Here, the method of combining both is a method in which a plurality of candidates are calculated by calculation, and the operator designates them from among them. Alternatively, this is a method in which an operator designates an area that is determined to include a plurality of feature points that are candidates for corresponding points, and the corresponding points are calculated by calculation in the designated area. Alternatively, the calculation may be performed by a method in which the entire imaging area is obtained without calculation by the operator. Hereinafter, a method for detecting corresponding points by calculation will be described.

(Corresponding point detection method 1)
First, an example in which corresponding points are automatically detected by stereo matching will be described. In this case, the image data of the measurement object obtained from the laser scanner 121 and the image data of the measurement object obtained from the digital camera 122 are compared, and the feature portions that coincide with each other are calculated as corresponding points. Here, the image data of the measurement object obtained from the laser scanner 121 is an image using pixel information as reflected light information (RGB intensity, etc.) of the three-dimensional point cloud position data acquired by the laser scanner 121. is there. Since the 3D point cloud position data has information related to the intensity of the reflected light, an image of the measurement object having the intensity of the reflected light as the grayscale information can be obtained by constructing an image using the information as pixel information. Can do.

Hereinafter, stereo matching will be described in detail. Stereo matching is a method in which the coordinate data of images in two coordinate systems are compared with each other, and the correspondence between the two images is obtained by the correlation between the two. In stereo matching, the correspondence between the feature points of the image viewed from each of the two viewpoints is obtained, and the corresponding points can be automatically extracted. FIG. 7 is an explanatory diagram for explaining the principle of stereo matching. In this method, as shown in the drawing, a template image of N ₁ × N ₁ pixel is moved on a search range (M ₁ −N ₁ +1) ² in an input image of M ₁ × M ₁ pixel larger than that, The upper left position of the template image is calculated so that the cross-correlation function C (a, b) expressed by the following equation 17 is maximized (that is, the degree of correlation is maximized).

  The above processing is performed while changing the magnification of one image and rotating it. And the area | region where both images correspond is calculated | required on the conditions from which the correlation became the maximum, Furthermore, the corresponding point is detected by extracting the feature point in this area | region.

  By using stereo matching, a matching part of two images to be compared can be specified, and the correspondence between the two coordinate systems can be known. In this method, the relative positional relationship between the two images is determined so as to maximize the correlation between the two images. The correlation between the two images is determined by the feature points of both images.

  When the corresponding point is detected from the image data based on the three-dimensional point cloud position data acquired by the laser scanner 121, the feature point to be detected is the intensity of reflected light (for example, RGB intensity), color, and hue of the irradiated laser light. , Saturation, or a combination of these. That is, when a feature point is characterized by the difference in reflected light intensity of each RGB color, when a feature point is characterized by a reflected light intensity of a specific color, or when characterizing a feature point by a difference in color (hue) of reflected light In some cases, a feature point is characterized by a difference in vividness of the color of reflected light, or a feature point is characterized by a combination thereof. Then, the feature point parameters (for example, RGB intensity) obtained on the laser scanner 121 side and the corresponding parameters (for example, RGB intensity) of the image captured by the digital camera 122 are compared by the above stereo matching, Corresponding points are detected.

  In addition, when performing stereo matching by comparing captured images, a part that is easily identified as a characteristic part (for example, an edge part of a structure) from the periphery in the captured image is treated as a characteristic point, and the corresponding point Is detected.

  Here, the case where image data based on the three-dimensional point cloud position data acquired by the laser scanner 121 is used as a comparison target has been described. However, the laser scanner 121 is a measurement target for acquiring the three-dimensional point cloud position data. Since it has a photographing means for photographing the object, image data obtained by photographing by this photographing means may be used. In this case, the above-described processing is performed based on the image data of the measurement object obtained by the imaging unit included in the laser scanner 121 and the image data of the measurement object obtained by the digital camera 122.

(Corresponding point detection method 2)
As a method for detecting the corresponding points, a method for comparing the parameters related to the three-dimensional shape of the measurement object and detecting the corresponding points based on the matching is given. Hereinafter, an example of this method will be described. As will be described later, the three-dimensional point cloud position data processing unit 108 performs processing that allows the appearance of the measurement object to be handled as a set of a plurality of surfaces based on the three-dimensional point cloud position data. Here, each surface is treated as a label that can be distinguished from the other. The label has surface position and surface orientation data. The position of the label is specified by, for example, the three-dimensional coordinates of the center of gravity of the surface.

  This label distribution state is compared by the same method as the stereo matching described above, and the positional relationship that maximizes the correlation is obtained, so that the two images of the measurement object configured by a plurality of labels coincide with each other. (Overlapping part) is detected. Then, by paying attention to this overlapping portion and extracting three-dimensional point cloud position data as feature points from the overlapped portion, the corresponding points of the two images can be specified. In order to extract feature points from the three-dimensional point cloud position data, a part where the non-planarity described later is partially remarkable will be described later (1) local curvature, (2) fitting accuracy to a local plane, (3) What is necessary is just to extract based on coplanarity.

  Hereinafter, an example of a method for matching two pieces of image data using the label will be described. FIG. 8 is a conceptual diagram illustrating a matching method using labels. FIG. 8A conceptually shows that the 3D point cloud position data is acquired from the first viewpoint for the measurement object, and further the 3D point cloud position data is acquired from the second viewpoint. Is described as a simplified model. Here, it is assumed that the first viewpoint is the acquisition position of the three-dimensional point cloud position data by the laser scanner 121, and the second viewpoint is the acquisition position of the image data obtained by photographing using the digital camera 122.

  Here, the measurement object is the first surface to which the label 1 is applied, the second surface to which the label 2 is applied, the third surface to which the label 3 is applied, and the fourth surface to which the label 4 is applied. It is comprised from the 4th surface to which the surface, the label 1 was provided, the 5th surface to which the label 5 was provided, and the 6th surface to which the label 6 was provided.

  Here, from the first viewpoint, the label 5 and the label 6 are occluded, and the three-dimensional point cloud position data of the portion is not acquired (of course, the data of the label 5 and the label 6 are not from the first viewpoint. (Not obtained) state is shown. FIG. 8B shows a labeled measurement object model (measurement object model constituted by labels) obtained from the first viewpoint. Originally, the labeled model of the measurement object is a three-dimensional model, but here, a two-dimensional simplified model is shown.

  Further, from the second viewpoint, the labels 1 and 2 are occluded, and the 3D point cloud position data of the part is not acquired (of course, the data of the labels 1 and 2 are obtained from the second viewpoint. (Not shown) state is shown. FIG. 8C shows a model of the labeled measurement object obtained from the second viewpoint.

  Before the alignment (that is, the identification of the correspondence) between the three-dimensional point cloud position data obtained from the two viewpoints, the labeling model of FIG. 8B and the labeling model of FIG. The correspondence is not clear. Here, by comparing the correlation between adjacent labels in the two labeling models and aligning the two labeling models based on the similarity, the labeling model in FIG. 8B and the labeling model in FIG. A method to clarify the correspondence of the labeling model is explained.

  The label is characterized by information on the direction of the labeled surface and the position of its center of gravity. Therefore, in the case of FIG. 8, the correlation between label 1 and label 2, the correlation between label 2 and label 3, and the correlation between label 3 and label 4 specify the relationship between the relative position and orientation of adjacent labels. To do. Therefore, the shape of the labeling model shown in FIG. 8B is specified by the correlation between the adjacent labels. That is, the three-dimensional structure of the labeling model of the measurement object having a three-dimensional shape can be specified by the correlation between adjacent labels in the labels constituting the labeling model.

  Using this, the two labeling models shown in FIGS. 8B and 8C are aligned. In this example, the combination of the label 3 and the label 4 (combination of the direction of the surface and the position of the center of gravity of the surface) is common in FIGS. 8B and 8C. Therefore, one or both of the labeling model of FIG. 8B and the labeling model of FIG. 8C are moved, and if necessary, the magnification is varied, and the parts of label 3 and label 4 having a common correlation By overlapping the two, it becomes possible to align the two labeling models. This state is shown in FIG.

  At this stage, since the digital camera 122 is not calibrated, the image based on the image captured by the digital camera 122 from the second viewpoint includes distortion, and the three-dimensional image obtained from the image is obtained. The accuracy of the point cloud position data includes an error. Therefore, the labeling model in FIG. 8C also has a larger error than the labeling model in FIG. 8B, and the matching shown in FIG. 8D includes an error (note that distortion in the lens system is the periphery of the screen). This error depends on the position in the screen where the alignment is performed). Therefore, an error threshold is determined in advance, and the consistency shown in FIG. 8D is determined based on the threshold. Alternatively, a method of calculating in advance an internal orientation element at a plurality of different focal lengths and calculating an internal orientation element at an arbitrary focal length from an approximate curve based on the internal orientation elements, which will be described later (see paragraphs “0144” and “0146”), is used. The lens distortion error here can be reduced in advance.

  As described above, the labeling model shown in FIG. 8C includes an error, but it is sufficiently possible to determine a shape match. That is, the method shown in FIG. 8 can sufficiently detect a portion where the shape of the measurement object grasped from the first viewpoint matches the shape of the measurement object grasped from the second viewpoint. The same applies to the comparison of three-dimensional shapes.

  By performing the alignment shown in FIG. 8D, a portion where the 3D point cloud position data obtained from the first viewpoint and the 3D point cloud position data obtained from the second viewpoint overlap (in this case) Indicates the portion of label 3 and label 4). Then, feature points are detected from the overlapping portions in the three-dimensional point cloud position data obtained by the laser scanner 121, and these portions are used as corresponding points. The feature points are detected based on (1) local curvature, (2) fitting accuracy to the local plane, and (3) coplanarity, which will be described later.

  The combination of labels to be compared is not limited to two adjacent ones as in the above example, and the same principle can be applied as long as there are a plurality of combinations. Further, the plurality of labels to be selected are separated from each other, and other labels may exist between them (for example, in the case of FIG. 8, selection of label 1 and label 3 is also possible).

  Here, an example of detecting corresponding points between two 3D point cloud position data using labels has been described, but 3D models created based on 3D point cloud position data are compared and matched. A technique for obtaining the part is also possible. That is, the first 3D model based on the 3D point cloud position data obtained from the first viewpoint is compared with the second 3D model based on the 3D point cloud position data obtained from the second viewpoint. Thus, by obtaining the matching three-dimensional shape portion, it is possible to obtain an overlapping portion between the three-dimensional point group position data obtained from the first viewpoint and the three-dimensional point group position data obtained from the second viewpoint.

  As will be described later with respect to the three-dimensional point cloud position data processing unit 108, the amount of data related to the label can be significantly reduced compared to the three-dimensional point cloud position data. Therefore, the calculation based on the principle shown in FIG. 8 can dramatically increase the calculation speed as compared with the case of performing matching using the three-dimensional point cloud position data as it is. The label information reflects information related to the three-dimensional shape of the measurement object. Therefore, it can be said that the identification of the correspondence relationship using the label is one of the methods for identifying the correspondence relationship between the two images by comparing information related to the three-dimensional shape of the measurement object. Since this method is based on comparison of information relating to a three-dimensional shape, the three-dimensional positional accuracy can be increased. Further, as described above, since the label data is significantly smaller than the three-dimensional point cloud position data, there is an advantage that the calculation time can be shortened and the burden on the arithmetic elements and the memory can be reduced.

(Internal orientation element calculation unit 106)
The internal orientation element calculation unit 106 calculates an internal orientation element of the digital camera 122. Here, based on the correspondence specified by the correspondence specifying unit 105, that is, the correspondence between the three-dimensional point cloud position data obtained using the laser scanner 121 and the three-dimensional point cloud position data obtained using the digital camera 122. The internal orientation element of the digital camera 122 is calculated. The operation for obtaining the internal orientation element is called calibration.

  In normal calibration, an internal orientation element of a camera is calculated based on a photographed image of a known three-dimensional field (a target arranged at a plurality of three-dimensional positions whose positions are clear). In the present embodiment, three-dimensional point cloud position data of the measurement object acquired by the laser scanner 121 is used instead of the above three-dimensional field. First, the three-dimensional point group position data obtained by the laser scanner 121 has the three-dimensional coordinates of each measurement point. On the other hand, a correspondence relationship between the three-dimensional point group position data obtained by the laser scanner 121 and the three-dimensional point group position data based on the captured image of the digital camera 122 is identified by the correspondence relationship identifying unit 105. Thereby, the three-dimensional coordinate of the specific point in the picked-up image of the digital camera 122 can be specified. Using this, calibration is performed using this specific point as a known target, thereby calculating the internal orientation element of the digital camera 122.

  That is, based on the three-dimensional point cloud position data acquired by the laser scanner 121, the three-dimensional coordinates of a certain point (usually a characteristic point that is characteristic in terms of image) in the image captured by the digital camera 122 are based on the above-described correspondence. It becomes clear. Here, a function or correction for correcting an image by performing processing (calibration) for obtaining an internal orientation element using a point (reference point) where the three-dimensional coordinates in the image captured by the digital camera 122 become clear as a target (reference point). Calculate the coefficient. This function and the correction coefficient are obtained at each grid point obtained by dividing the screen into a grid.

Hereinafter, an example of an internal orientation element calculation method performed by the internal orientation element calculation unit 106 will be described. First, consider the photographic coordinates (x, y) of a certain point in a photographic image. Here, it is assumed that the photographic coordinates (x, y) include an error caused by distortion of the optical system and the like, and correction terms for correcting them are Δx and Δy. Here, when a correction model in consideration of distortion in the radial direction and the tangential direction of the lens is applied, Δx and Δy are expressed by the following equation (18). Here, K _{1 to} K ₂ and P _{1 to} P ₂ are parameters reflecting the internal orientation elements.

Here, since the three-dimensional coordinates of the target in the photographed photograph are known, unknown parameters K _{1 to} K ₂ and P _{1 to} P ₂ are obtained based on the above polynomial when attention is paid to a large number of targets. As a result, the image captured by the digital camera 122 can be corrected.

(Image correction unit 107)
The image correction unit 107 corrects the image captured by the digital camera 122 based on the internal orientation element calculated by the internal orientation element calculation unit 106. This correction is performed at lattice points obtained by dividing the target image into a lattice shape, and distortion of the captured image caused by the optical system of the digital camera 122 is eliminated (or reduced). Specifically, the coordinate value calculated by the coordinate calculation unit 104 is corrected based on the internal orientation element inside the digital camera 122. Thereby, for example, the deviation of the three-dimensional coordinate data of the photographing object due to the distortion of the lens system is corrected, and the accuracy of the three-dimensional point group position data of the measuring object calculated based on the stereo method can be improved.

(3D point cloud position data processing unit 108)
The three-dimensional point cloud position data processing unit 108 calculates non-surface regions, removes non-surface regions, labeling processing, calculates feature parts such as contour lines, creates a three-dimensional model composed of contour lines, and Perform related operations. The three-dimensional model is an image obtained by visualizing the three-dimensional structure of the measurement object in which the outline of the measurement object is expressed as a diagram. The outline is an outline that forms the outline of the measurement object, which is necessary for visually grasping the appearance of the measurement object. Specifically, a bent portion or a portion where the curvature is rapidly reduced becomes a contour line. The contour line is not limited to the outer contour part, but the edge part that characterizes the protruding part of the convex part, or the edge that characterizes the concave part (for example, the part of the groove structure). The part of is also the target. A so-called diagram can be obtained from the contour line, and an image can be displayed so that the appearance of the object can be easily grasped. Note that the three-dimensional model includes not only the above-described line information but also information on points that are characteristic portions when the appearance of the measurement object is visually grasped.

  Details of the three-dimensional point cloud position data processing unit 108 in FIG. 1 will be described below. FIG. 2 shows a block diagram of the three-dimensional point cloud position data processing unit 108. The three-dimensional point cloud position data processing unit 108 includes a non-surface area calculation unit 201, a non-surface area removal unit 202, a surface labeling unit 203, and a contour line calculation unit 204. Hereinafter, each of these functional units will be described. The non-surface region calculation unit 201 includes a local region acquisition unit 201a that acquires a local region, a normal vector calculation unit 201b that calculates a normal vector of the local region, a local curvature calculation unit 201c that calculates a local curvature of the local region, A local plane calculation unit 201d that calculates a local plane to be fitted to the region is provided.

  The local area acquisition unit 201a acquires, as a local area, a square area (grid-like area) having about 3 to 7 pixels on a side centered on the point of interest based on the three-dimensional point cloud position data. The normal vector calculation unit 201b calculates a normal vector of each point in the local region acquired by the local region acquisition unit 201a. In the process of calculating the normal vector, attention is paid to the three-dimensional point cloud position data in the local region, and the normal vector of each point is calculated. This process is performed for all three-dimensional point cloud position data. That is, the three-dimensional point cloud position data is divided into countless local regions, and the normal vector of each point is calculated in each local region.

  The local curvature calculation unit 201c calculates variation (local curvature) of normal vectors in the above-described local region. Here, the average (mNVx, mNVy, mNVz) of the intensity values (NVx, NVy, NVz) of the three-axis components of each normal vector is obtained in the local region of interest, and the standard deviation (StdNVx, StdNVy, StdNVz) ) Next, the square root of the square sum of the standard deviation is calculated as a local curvature (crv) (Equation 19).

  The local plane calculation unit 201d obtains a local plane that is fitted (approximated) to the local region. In this process, an equation of the local plane is obtained from the three-dimensional coordinates of each point in the local region of interest. The local plane is a plane that is fitted to the local region of interest. Here, the equation of the surface of the local plane to be fitted to the local region is calculated using the least square method. Specifically, a plurality of different plane equations are obtained, compared with each other, and a surface equation of the local plane to be fitted to the local region is calculated. If the local region of interest is a plane, the local plane and the local region match. The above processing is repeated so that all the three-dimensional point cloud position data are targeted while sequentially shifting the local area, and the normal vector, local plane, and local curvature in each local area are obtained.

  The non-surface area removing unit 202 performs a process of removing points in the non-surface area based on the normal vector, local plane, and local curvature in each local area obtained above. That is, in order to extract a surface (a plane and a curved surface), a portion (non-surface region) that can be determined not to be a surface in advance is removed. The non-surface region is a region that is neither a plane nor a curved surface, but may include a curved surface with a high curvature depending on the following threshold values (1) to (3).

  The non-surface area removal unit 202 removes the calculated non-surface area 3D point group position data from the 3D point group position data acquired by the 3D point group position data acquisition unit 101. The non-surface area removal process can be performed using at least one of the following three methods. Here, the determination by the following methods (1) to (3) is performed for all the local regions described above, and the local region determined as the non-surface region by one or more methods is configured as the non-surface region. Extract as a local region. Then, the 3D point cloud position data relating to the points constituting the extracted non-surface area is removed.

(1) A portion having a high local curvature: The above-described local curvature is compared with a preset threshold value, and a local region having a local curvature exceeding the threshold value is determined as a non-surface region. Since the local curvature represents the variation of the normal vector at the point of interest and its peripheral points, the value is small for a surface (a flat surface and a curved surface with a small curvature), and the value is large for a surface other than a surface (non-surface). Therefore, if the local curvature is larger than a predetermined threshold, the local region is determined as a non-surface region.

(2) Fitting accuracy to the local plane: When the distance between each point of the local area and the corresponding local plane is calculated and the average of these distances is greater than a preset threshold, the local area is determined as a non-surface area. judge. That is, when the local area is in a state of being deviated from the plane, the distance between the local plane corresponding to each point of the local area increases as the degree becomes more severe. Using this fact, the degree of non-surface of the local region is determined.

(3) Coplanarity check: Here, the directions of corresponding local planes are compared in adjacent local regions. If the difference in orientation of the local planes exceeds the threshold value, it is determined that the local area to be compared belongs to the non-plane area. Specifically, if the inner product of the normal vector of the two local planes fitted to each of the two target local regions and the vector connecting the center points is 0, both local planes are on the same plane. Is determined to exist. Moreover, it is determined that the extent that the two local planes are not on the same plane is more remarkable as the inner product becomes larger.

  In the determination by the above methods (1) to (3), a local region determined as a non-surface region by one or more methods is extracted as a local region constituting the non-surface region. Then, the 3D point cloud position data relating to the points constituting the extracted local region is removed from the 3D point cloud position data to be calculated. The non-surface area is removed as described above. Since the removed 3D point cloud position data may be used in later processing, it is stored in an appropriate storage area so that it can be identified from the 3D point cloud position data that has not been removed. To make it available for later use.

  Next, the function of the surface labeling unit 203 will be described. The surface labeling unit 203 performs surface labeling on the 3D point cloud position data from which the 3D point cloud position data of the non-surface area is removed based on the continuity of normal vectors. Specifically, if the angle difference between the normal vector of a specific point of interest and an adjacent point is less than a predetermined threshold, the same label is attached to those points. By repeating this operation, the same label is attached to a continuous flat surface and a continuous gentle curved surface, and these can be identified as one surface. Further, after the surface labeling, it is determined whether the label (surface) is a plane or a curved surface with a small curvature by using the angle difference between the normal vectors and the standard deviation of the three-axis components of the normal vectors. Identification data for identifying the fact is associated with each label.

  Subsequently, the label (surface) having a small area is removed as noise. This noise removal may be performed simultaneously with the surface labeling process. In this case, while performing surface labeling, the number of points of the same label (the number of points constituting the label) is counted, and a process of canceling the label having the number of points equal to or less than a predetermined value is performed. Next, the same label as the nearest surface (closest surface) is given to the point having no label at this time. As a result, the already labeled surface is expanded.

  That is, the equation of the surface with the label is obtained, and the distance between the surface and the point without the label is obtained. If there are a plurality of labels (surfaces) around a point where there is no label, the label with the shortest distance is selected. If there is still a point with no label, the threshold values for non-surface area removal, noise removal, and label expansion are changed, and related processing is performed again. For example, in non-surface area removal, the number of points to be extracted as non-surface is reduced by increasing the threshold value of local curvature. Alternatively, in label expansion, by increasing the threshold of the distance between a point without a label and the nearest surface, more labels are given to the point without a label.

  Next, even if the labels are different surfaces, the labels are integrated when they are the same surface. In this case, even if the surfaces are not continuous, the same label is attached to the surfaces having the same position or orientation. Specifically, by comparing the position and orientation of the normal vector of each surface, the same non-continuous surface is extracted and unified to the label of any surface. The above is the function of the surface labeling unit 203.

  According to the function of the surface labeling unit 203, the amount of data to be handled can be compressed, so that the processing of the three-dimensional point cloud position data can be speeded up. In addition, the required amount of memory can be saved. In addition, it is possible to remove the three-dimensional point cloud position data of a passerby or a vehicle that has passed through during measurement as noise.

  The contour calculation unit 204 calculates (estimates) the contour based on the three-dimensional point cloud position data of the adjacent surfaces. Hereinafter, a specific calculation method will be described. The contour line calculation unit 204 obtains an intersection line between adjacent surfaces sandwiching a non-surface region therebetween, and performs a process of using this as a contour line. At this time, a method of approximating the non-planar region with a plurality of local planes by fitting a local plane to a non-planar region between adjacent surfaces and connecting a plurality of local planes may be employed. This can be regarded as an approximation of a non-planar region by a polyhedron composed of a plurality of local planes. In this case, the local planes are connected from the adjacent surfaces, and the line of intersection between the adjacent local planes is calculated as the contour line. By calculating the contour line, the contour image of the measurement object becomes clear.

  Next, the two-dimensional edge calculation unit 205 will be described. Hereinafter, an example of processing performed by the two-dimensional edge calculation unit 205 will be described. First, based on the intensity distribution of the reflected light from the object, a region of a two-dimensional image corresponding to a segmented (segmented) surface using a known edge extraction operator such as Laplacian, Pleuwit, Sobel, Canny Extract edges from within. In other words, since the two-dimensional edge is recognized by the difference in shading in the surface, the difference in shading is extracted from the information on the intensity of the reflected light, and by setting a threshold value in the extraction condition, the border between the shading is used as an edge. Extract. Next, the height (z value) of the three-dimensional coordinates of the points constituting the extracted edge, and the height (z value) of the three-dimensional coordinates of the points constituting the neighboring contour line (three-dimensional edge) If the difference is within a predetermined threshold, the edge is extracted as a two-dimensional edge. That is, it is determined whether or not a point constituting the edge extracted on the two-dimensional image is on the segmented surface, and if it is determined that the point is on the surface, it is set as the two-dimensional edge.

  After the calculation of the two-dimensional edge, the contour line calculated by the contour calculation unit 204 and the two-dimensional edge calculated by the two-dimensional edge calculation unit 205 are integrated. This process is performed in the edge integration unit 206. Thereby, edge extraction based on the three-dimensional point cloud position data is performed. By extracting the edge, a line (contour line) constituting the appearance of the measurement object when the measurement object is visually recognized is extracted. Based on the obtained edge information, the three-dimensional model forming unit 207 forms a three-dimensional model (a diagram image) of the measurement object. Hereinafter, a specific example of the three-dimensional model will be described. For example, a case will be described in which a building is selected as an object to be measured and a three-dimensional model is obtained based on the three-dimensional point cloud position data of the building. In this case, a three-dimensional model in which the outline of the building, the pattern of the outer wall, the outline of the window frame and the like is represented by a diagram is obtained.

(Operation example)
An example of a specific operation of the optical information processing apparatus 100 in FIG. 1 will be described. FIG. 9 is a flowchart illustrating an example of a processing procedure. FIG. 10 conceptually shows a situation where the processing described here is performed. FIG. 10 shows a building 120 to be measured, a passenger car 121 parked in front of the building 120, a laser scanner 122, a personal computer 123, and a stereo pair image capturing device 124.

  The laser scanner 122 corresponds to the laser scanner 121 of FIG. 1 and acquires the three-dimensional point cloud position data of the building 120 that is the measurement object. Here, the laser scanner 122 is arranged at the first viewpoint. The personal computer 123 has the function of the optical information processing apparatus 100 of FIG. 1 and is connected to two digital cameras, a laser scanner 122 and a stereo pair image photographing apparatus 124, and the three-dimensional point cloud position data from the laser scanner 122 and Image data from the two digital cameras 125 and 126 is input. The stereo pair image photographing device 124 corresponds to the digital camera of FIG. 1 and has a structure in which two digital cameras 125 and 126 are arranged on a dedicated stand 127 at a distance from each other. Is used to photograph the building 120 as a measurement object, and image data of the stereo pair image is acquired. Here, the stereo pair image capturing device 124 is arranged at a second viewpoint different from the first viewpoint at which the laser scanner 122 is installed.

  FIG. 10 shows a state in which occlusion occurs when a part of the building 120 as a measurement object is shaded by the passenger car 121 parked in front when viewed from the first viewpoint. Therefore, the three-dimensional point cloud position data of the building 120 obtained by the laser scanner 122 at the first viewpoint has a defect due to the occlusion. In this example, a case where the second viewpoint is selected as a position to eliminate this occlusion is shown.

  Hereinafter, an example of a processing procedure will be described with reference to FIG. When the processing is started (step S901), first, the laser scanner 122 is arranged at the first viewpoint, and from there, the three-dimensional point cloud position data of the building 120 (first three-dimensional point cloud position using the laser beam). Data) is acquired (step S902). Next, the three-dimensional point cloud position data of the building 120 acquired in step S902 is sent to the personal computer 123 and processed by the three-dimensional point cloud position data processing unit 108 shown in FIGS. 1 and 2 (step S903). In this process, a three-dimensional model of the building 120 and intermediate data (data related to a non-surface area and label data) obtained in the process are acquired.

  After step S903, the stereo pair image capturing device 124 is installed at the second viewpoint in FIG. 10, and the stereo pair image of the building 120 is captured from the second viewpoint (step S904). The image data of the stereo pair image taken here is sent to the personal computer 123. The personal computer 123 that has received the image data of the stereo pair image calculates the 3D point cloud position data (second 3D point cloud position data) of the building 120 based on the stereo pair image (step S905). . This processing is performed in the coordinate calculation unit 104 in FIG.

  Next, a second tertiary obtained based on the first three-dimensional point cloud position data obtained by the laser scanner 122 at the first viewpoint and the stereo pair image photographed by the stereo pair image photographing device 124 at the second viewpoint. Corresponding points between the original point cloud position data are designated (step S906), and the correspondence between the three-dimensional point cloud position data based on the designated corresponding points is calculated (step S907). These processes are performed in the correspondence relationship specifying unit 103 in FIG.

  After specifying the correspondence between the two three-dimensional point cloud position data, it was selected from the three-dimensional point cloud position data (first three-dimensional point cloud position data) acquired by the laser scanner 122 at the first viewpoint. Calibration is performed with a point (reference point) as a point that is an image-like characteristic portion, and the internal orientation elements of the digital cameras 125 and 126 are calculated (step S908). This process is performed in the internal orientation element calculation unit 106 in FIG.

  Next, based on the calculated internal orientation element, the stereo pair image captured by the stereo pair image capturing device 124 is corrected at the second viewpoint (step S909). This processing is performed by the image correction unit 107 in FIG.

  Next, based on the stereo pair image corrected in step S909, the 3D point cloud position data (second 3D point cloud position data) of the building 120 as the measurement object is recalculated (step S910). This processing is performed by the coordinate calculation unit 104 in FIG. In this process, since the influence of image distortion caused by lens aberration or the like is corrected in step S909, three-dimensional point group position data in which the error caused by lens aberration or the like is reduced or eliminated is calculated. The

  Next, the correspondence relationship between the first three-dimensional point group position data acquired in step S902 and the second three-dimensional point group position data calculated again in step S910 is calculated again (step S911). This processing is performed in the correspondence relationship specifying unit 105 in FIG. At this time, since the accuracy of the second three-dimensional point cloud position data is improved by the process of step S910 from that at the time of step S907, a more accurate correspondence can be obtained.

  Next, based on the correspondence obtained in step S911, the three-dimensional point cloud position data of the building 120 acquired using the laser scanner 122 at the first viewpoint in FIG. 10 (the first three-dimensional point obtained in step S902). Group position data) and the three-dimensional point cloud position data of the building 120 obtained based on the stereo pair image of the building 120 photographed by the stereo pair image photographing device 124 at the second viewpoint (the three-dimensional point cloud obtained in step S910). Position data) and a three-dimensional model of the building 120 is created based on the integrated three-dimensional point cloud position data. Since this 3D model is complemented by 3D point cloud position data based on the stereo pair image obtained from the 2nd viewpoint, the 3D point cloud position data of the part that is occluded by the passenger car from the 1st viewpoint. Occlusion caused by the passenger car 121 is eliminated. The stereo pair image capturing device 124 has been described as an apparatus for obtaining the second viewpoint. However, two or more images may be captured by a single digital camera.

  In the processing described above, laser scanning of the building 120 is performed from the first viewpoint to obtain three-dimensional point cloud position data. On the other hand, a stereo pair image including the part of the building 120 that is occluded from the second viewpoint to the first viewpoint is captured. Then, by obtaining the correspondence between the stereo image obtained from the second viewpoint and the three-dimensional point cloud position data obtained from the first viewpoint, the stereo pair image shooting using the three-dimensional point cloud position data is obtained. The device 124 is calibrated.

  According to this processing, since the acquisition of the three-dimensional point cloud position data is not performed again in order to eliminate the occlusion, the labor required to acquire the point cloud position data again does not occur. In addition, since the digital cameras 125 and 126 can be calibrated from the second viewpoint, the second viewpoint can be freely selected. In addition, since the calibration at the second viewpoint is performed based on the three-dimensional point cloud position data already acquired by the laser scanner 122 at the first viewpoint, target setting and distance setting necessary for calibration are performed. Such a complicated work is not required.

(Other)
In the second viewpoint of FIG. 10, the building 120 is divided into a plurality of parts so as to partially overlap, and stereo pair images are taken, and further, a plurality of sets of stereo pair images are obtained in steps S904 to S904. There is also a method of performing step S908. According to this method, the accuracy of the calculation result can be increased by repeated calculation.

  Approximation that shows the relationship between the focal length of the camera and the internal orientation element based on the internal orientation elements obtained at different focal lengths in advance, as a further development of the method for acquiring the internal orientation element of the camera at a fixed focus There is a method of calculating an internal orientation element at an arbitrary focal length by using a curve. In this method, the above approximate curve data is stored as focal length information in a storage unit inside the camera or an appropriate storage medium, and internal orientation is determined based on the focal length information and the distance to the object under the shooting conditions. The element is calculated. In this case, although it is an estimated value or an approximate value, an internal orientation element in the photographing condition can be obtained. Calibration using the present invention can be combined with this method.

  That is, the above-described method of calculating the internal orientation element corresponding to the focal length from the approximate curve includes an error, but a certain range of accuracy can be obtained from an accurate value. Therefore, in the calculation of the internal orientation element performed in step S908, the value of the internal orientation element obtained from the approximate curve can be used as a reference value, whereby the calculation can be made more efficient. For example, in the calculation performed in step S908, an operation for converging the solution by performing approximate calculation or iterative calculation is required. If a value serving as a guide at this time is known in advance, the guide is used. Since the calculation can be performed on the assumption that a solution exists (or converges) near the value, the labor of calculation may be further reduced as compared with the case where there is no guideline. Since the calculation is labor-saving, the computation time of the personal computer 123 is shortened. In addition, since the approximate value of the solution is known in advance, it is possible to avoid the inconvenience that the solution does not converge and an error occurs. As a form for obtaining an internal orientation element corresponding to the focal length from an approximate curve, there is a form in which the approximate focal length is recorded in advance in a camera file when a zoom lens or an autofocus lens is used.

  It is also effective to correct the stereo pair image captured in step S904 by using the approximate curve in the image correction unit 107 and increase the accuracy. In this case, since the accuracy of the stereo pair image obtained in step S904 can be increased, the calculation of the second three-dimensional point cloud position data in step S905 based on the stereo pair image obtained in step S904 can be reduced in advance. . By doing in this way, the labor of calculation is further saved and it becomes easy to obtain an accurate result.

2. Second Embodiment Hereinafter, a three-dimensional laser scanner having a function of processing three-dimensional point cloud position data will be described. In this example, the three-dimensional laser scanner irradiates the object to be measured with distance measuring light (laser light), and performs a number of measurements on the object to be measured from its own position based on the time of flight of the laser light. Measure the distance to the point. The three-dimensional laser scanner detects the irradiation direction (horizontal angle and elevation angle) of the laser beam, and calculates the three-dimensional coordinates of the measurement point based on the distance and the irradiation direction. The three-dimensional laser scanner acquires a two-dimensional image (RGB intensity at each measurement point) obtained by photographing the measurement object, and forms three-dimensional point group position data that combines the two-dimensional image and the three-dimensional coordinates. . Furthermore, the three-dimensional laser scanner shown here has a function of performing the processing of the optical information processing apparatus 100 described with reference to FIG.

(Constitution)
11 and 12 are cross-sectional views showing the configuration of the three-dimensional laser scanner 1. The three-dimensional laser scanner 1 includes a leveling unit 22, a rotation mechanism unit 23, a main body unit 27, and a rotation irradiation unit 28. The main body 27 includes a distance measuring unit 24, a photographing unit 25, a control unit 26, and the like. For convenience of explanation, FIG. 11 shows a state where only the rotary irradiation unit 28 is viewed from the side with respect to the cross-sectional direction shown in FIG.

  The leveling unit 22 has a base plate 29, and the rotation mechanism unit 23 has a lower casing 30. The lower casing 30 is supported on the base plate 29 at three points by a pin 31 and two adjustment screws 32. The lower casing 30 tilts with the tip of the pin 31 as a fulcrum. A tension spring 33 is provided between the base plate 29 and the lower casing 30 to prevent the base plate 29 and the lower casing 30 from separating from each other.

  Two leveling motors 34 are provided inside the lower casing 30. The two leveling motors 34 are driven by the control unit 26 independently of each other. When the leveling motor 34 is driven, the adjustment screw 32 is rotated via the leveling drive gear 35 and the leveling driven gear 36, and the amount of downward protrusion of the adjustment screw 32 is adjusted. An inclination sensor 37 (see FIG. 13) is provided inside the lower casing 30. The two leveling motors 34 are driven by the detection signal of the tilt sensor 37, whereby leveling is executed.

  The rotation mechanism unit 23 includes a horizontal angle drive motor 38 inside the lower casing 30. A horizontal rotation drive gear 39 is fitted to the output shaft of the horizontal angle drive motor 38. The horizontal rotation drive gear 39 is meshed with the horizontal rotation gear 40. The horizontal rotation gear 40 is provided on the rotation shaft portion 41. The rotating shaft portion 41 is provided at the center portion of the rotating base 42. The rotating base 42 is provided on the upper portion of the lower casing 30 via a bearing member 43.

  Further, the rotary shaft portion 41 is provided with, for example, an encoder as the horizontal angle detector 44. The horizontal angle detector 44 detects a relative rotation angle (horizontal angle) of the rotation shaft portion 41 with respect to the lower casing 30. The horizontal angle is input to the control unit 26, and the control unit 26 controls the horizontal angle drive motor 38 based on the detection result.

  The main body 27 has a main body casing 45. The main body casing 45 is fixed to the rotating base 42. A lens barrel 46 is provided inside the main body casing 45. The lens barrel 46 has a rotation center concentric with the rotation center of the main body casing 45. The center of rotation of the lens barrel 46 is aligned with the optical axis 47. Inside the lens barrel 46, a beam splitter 48 as a light beam separating means is provided. The beam splitter 48 has a function of transmitting visible light and reflecting infrared light. The optical axis 47 is separated into an optical axis 49 and an optical axis 50 by a beam splitter 48.

  The distance measuring unit 24 is provided on the outer periphery of the lens barrel 46. The distance measuring unit 24 includes a pulse laser light source 51 as a light emitting unit. Between the pulse laser light source 51 and the beam splitter 48, a perforated mirror 52 and a beam waist changing optical system 53 for changing the beam waist diameter of the laser light are arranged. The distance measuring light source unit includes a pulse laser light source 51, a beam waist changing optical system 53, and a perforated mirror 52. The perforated mirror 52 has a role of guiding the pulsed laser light from the hole 52 a to the beam splitter 48, and reflecting the reflected laser light returned from the measurement object toward the distance measuring light receiving unit 54.

  The pulse laser light source 51 emits infrared pulse laser light at a predetermined timing under the control of the control unit 26. The infrared pulse laser beam is reflected by the beam splitter 48 toward the high / low angle rotating mirror 55. The elevation mirror 55 for high and low angles reflects the infrared pulse laser beam toward the measurement object. The elevation mirror 55 is rotated in the elevation direction to convert the optical axis 47 extending in the vertical direction into a projection optical axis 56 in the elevation direction. A condensing lens 57 is disposed between the beam splitter 48 and the elevation mirror 55 and inside the lens barrel 46.

  The reflected laser light from the object to be measured is guided to the distance measuring light receiving unit 54 through the elevation angle turning mirror 55, the condenser lens 57, the beam splitter 48, and the perforated mirror 52. Further, the reference light is also guided to the distance measuring light receiving unit 54 through the internal reference light path. Based on the difference between the time until the reflected laser light is received by the distance measuring light receiving unit 54 and the time until the laser light is received by the distance measuring light receiving unit 54 through the internal reference light path, the optical information processing apparatus 1 The distance from the object to be measured (measurement target point) is measured. The ranging light receiving unit 54 is configured by a photoelectric change element such as a CMOS optical sensor, and also has a function of detecting the RGB intensity of the detected light.

  The photographing unit 25 includes an image light receiving unit 58 and functions as a camera. The image light receiving unit 58 is provided at the bottom of the lens barrel 46. The image light receiving unit 58 is configured by a pixel in which a large number of pixels are arranged in a plane, for example, a CCD (Charge Coupled Device). The position of each pixel of the image light receiving unit 58 is specified by the optical axis 50. For example, an XY coordinate is assumed with the optical axis 50 as the origin, and a pixel is defined as a point of the XY coordinate.

  The rotary irradiation unit 28 is accommodated in the light projection casing 59. A part of the peripheral wall of the light projection casing 59 serves as a light projection window. As shown in FIG. 12, a pair of mirror holder plates 61 are provided facing the flange portion 60 of the lens barrel 46. A rotation shaft 62 is stretched over the mirror holder plate 61. The high / low angle turning mirror 55 is fixed to the turning shaft 62. An elevation gear 63 is fitted to one end of the rotation shaft 62. An elevation angle detector 64 is provided on the other end side of the rotation shaft 62. The elevation angle detector 64 detects the rotation angle of the elevation angle rotation mirror 55 and outputs the detection result to the control unit 26.

  A high and low angle drive motor 65 is attached to one side of the mirror holder plate 61. A drive gear 66 is fitted on the output shaft of the high / low angle drive motor 65. The drive gear 66 is meshed with an elevation gear 63 attached to the rotary shaft 62. The elevation motor 65 is appropriately driven by the control of the control unit 26 based on the detection result of the elevation detector 64.

  On the upper part of the light projection casing 59, there is provided an illumination star turret 67. The sight sight gate 67 is used for roughly collimating the measurement object. The collimation direction using the sight sight gate 67 is a direction orthogonal to the direction in which the projection light axis 56 extends and the direction in which the rotation shaft 62 extends. In addition, as shown in FIG. 12, a GPS antenna 81 is disposed on the upper part of the light projection casing 59. GPS information is acquired by the GPS antenna, and the GPS information can be used for calculations performed internally.

  FIG. 13 is a block diagram of the control unit. Detection signals from the horizontal angle detector 44, the elevation angle detector 64, the tilt sensor 37, and the GPS antenna 81 are input to the control unit 26. The control unit 26 receives an operation instruction signal from the operation unit 6. The control unit 26 drives and controls the horizontal angle drive motor 38, the elevation angle drive motor 65, and the leveling motor 34, and controls the display unit 7 that displays the work status, measurement results, and the like. An external storage device 68 such as a memory card or HDD can be attached to and detached from the control unit 26.

  The control unit 26 includes a calculation unit 4, a storage unit 5, a horizontal drive unit 69, a height drive unit 70, a leveling drive unit 71, a distance data processing unit 72, an image data processing unit 73, and the like. The storage unit 5 includes a sequence program, a calculation program, a measurement data processing program for executing measurement data processing, an image processing program for performing image processing, and a three-dimensional point, which are necessary for distance measurement and detection of elevation angle and horizontal angle. A program for extracting a surface from group position data and further calculating a contour line, an image display program for displaying the calculated contour line on the display unit 7, and a process related to reacquisition of three-dimensional point group position data are controlled. In addition to storing various programs such as programs, an integrated management program or the like for integrating and managing these various programs is stored. The storage unit 5 stores various data such as measurement data and image data. The horizontal drive unit 69 drives and controls the horizontal angle drive motor 38, the elevation drive unit 70 controls the drive of the elevation angle drive motor 65, and the leveling drive unit 71 controls the leveling motor 34. The distance data processing unit 72 processes the distance data obtained by the distance measuring unit 24, and the image data processing unit 73 processes the image data obtained by the photographing unit 25.

  The control unit 26 includes a GPS receiving unit 82. The GPS receiver 82 processes a signal from a GPS satellite received by the GPS antenna, and calculates coordinate data on the earth. This is the same as a normal GPS receiver. Position information obtained from the GPS is input to the optical information processing unit 100 ′.

  FIG. 14 is a block diagram of the calculation unit 4. The calculation unit 4 includes a three-dimensional coordinate calculation unit 74, a link formation unit 75, a grid formation unit 9, and an optical information processing unit 100 '. The three-dimensional coordinate calculation unit 74 receives the distance data of the measurement target point from the distance data processing unit 72, and the direction data (horizontal angle and elevation angle) of the measurement target point from the horizontal angle detector 44 and the elevation angle detector 64. Is entered. Based on the input distance data and direction data, the three-dimensional coordinate calculation unit 74 calculates the three-dimensional coordinates (orthogonal coordinates) of each measurement point with the position of the optical information processing apparatus 1 as the origin (0, 0, 0). calculate.

  The link forming unit 75 receives the image data from the image data processing unit 73 and the coordinate data of the three-dimensional coordinates of each measurement point calculated by the three-dimensional coordinate calculation unit 74. The link forming unit 75 forms 3D point group position data 2 in which image data (RGB intensity at each measurement point) and 3D coordinates are linked. That is, when focusing on a point on the measurement object, the link forming unit 75 creates a link in which the position of the point of interest in the two-dimensional image is associated with the three-dimensional coordinates of the point of interest. The associated data is calculated for all the measurement points, and becomes the three-dimensional point group position data 2.

  The link forming unit 75 outputs the above three-dimensional point cloud position data 2 to the grid forming unit 9. When the distance between adjacent points in the three-dimensional point cloud position data 2 is not constant, the grid forming unit 9 forms an equally spaced grid (mesh) and registers the point closest to the grid intersection. Or the grid formation part 9 correct | amends all the points to the intersection position of a grid using a linear interpolation method or a bicubic method. If the distance between the points in the 3D point cloud position data 2 is constant, the processing of the grid forming unit 9 can be omitted.

Hereinafter, the grid formation procedure will be described. FIG. 15 is a conceptual diagram showing three-dimensional point group position data in which the distance between points is not constant, and FIG. 16 is a conceptual diagram showing a formed grid. As shown in FIG. 16, the average horizontal interval H ₁ to _N of each column is obtained, the difference ΔH _{i, j} of the average horizontal interval between the columns is calculated, and the average is set as the horizontal interval ΔH of the grid (Equation 20 ). The vertical interval is calculated by calculating the distance ΔV _{N, H} between the vertical adjacent points in each column and the average of ΔV _{N, H in} the entire image of the image sizes W, H is defined as the vertical interval ΔV (Equation 21 ). Then, as shown in FIG. 16, a grid with the calculated horizontal interval ΔH and vertical interval ΔV is formed.

  Next, the point closest to the intersection of the formed grids is registered. At this time, a predetermined threshold is provided for the distance from the intersection to each point to limit registration. For example, the threshold value is ½ of the horizontal interval ΔH and the vertical interval ΔV. It should be noted that all points may be corrected by applying a weight according to the distance from the intersection, such as a linear interpolation method or a bicubic method. However, when interpolation is performed, the point is not originally measured.

  The three-dimensional point cloud position data obtained as described above is output to the optical information processing unit 100 '. The optical information processing unit 100 ′ performs the same operation as that of the optical information processing apparatus 100 described in the first embodiment. Further, the display of the image presented to the user in the operation is displayed on the display unit 7 which is a liquid crystal display. The optical information processing unit 100 ′ is hardware having the same function as that of the optical information processing apparatus 100 of FIG. 1, and is configured by a dedicated integrated circuit using FPGA.

  Coordinate data on the earth obtained from the GPS receiving unit 82 is input to the optical information processing unit 100 ′. According to this configuration, coordinates handled by the optical information processing unit 100 ′ are linked to position data (for example, electronic map information) obtained from the GPS. Thereby, for example, the installation position of the three-dimensional laser scanner 1 can be displayed on the screen on the electronic map.

(Other)
In the configuration of the control unit 26, assuming that the three-dimensional point cloud position data is output from the grid forming unit 9, the apparatus shown in FIGS. 11 and 12 uses the personal computer shown in the first embodiment. The three-dimensional laser scanner can be used in combination with the optical information processing apparatus. A configuration in which the processing performed by the optical information processing unit 100 ′ is performed in a distributed manner is also possible. For example, a configuration in which a part of the functions of the optical information processing unit 100 ′ is performed by a server connected by a communication line is also possible. The same applies to the optical information processing apparatus 100 of FIG. These configurations are understood as an example of the optical information processing system of the present invention.

  As a method for acquiring an image, a method by photographing using a CCD camera or the like is common, but an image of a measurement object can also be reproduced based on point cloud data. When three-dimensional point cloud position data is obtained by a laser scanner, data relating to the light intensity of reflected light from each point is obtained. Therefore, by treating the 3D point cloud position data as pixel data constituting the image of the object, the image of the measurement object can be reproduced based on the 3D point cloud position data. That is, an image of the measurement object can be obtained using a laser scanner instead of a photographing means such as a CCD or a CMOS image sensor. In this case, the image receiving unit 102 in FIG. 1 acquires image data based on the above-described principle based on the three-dimensional point cloud position data output from the laser scanner 121.

  The present invention can be used in a technique for measuring three-dimensional information.

  DESCRIPTION OF SYMBOLS 1 ... Three-dimensional laser scanner, 2 ... Three-dimensional point cloud position data, 22 ... Leveling part, 23 ... Rotation mechanism part, 24 ... Distance measuring part, 25 ... Imaging | photography part, 26 ... Control part, 27 ... Main-body part, 28 DESCRIPTION OF SYMBOLS ... Rotary irradiation part, 29 ... Base, 30 ... Lower casing, 31 ... Pin, 32 ... Adjustment screw, 33 ... Tension spring, 34 ... Leveling motor, 35 ... Leveling drive gear, 36 ... Leveling driven gear, 37 DESCRIPTION OF SYMBOLS ... Inclination sensor, 38 ... Horizontal rotation motor, 39 ... Horizontal rotation drive gear, 40 ... Horizontal rotation gear, 41 ... Rotating shaft part, 42 ... Rotary base, 43 ... Bearing member, 44 ... Horizontal angle detector, 45 ... Main body casing, 46 ... Tube, 47 ... Optical axis, 48 ... Beam splitter, 49, 50 ... Optical axis, 51 ... Pulse laser light source, 52 ... Perforated mirror, 53 ... Beam waist changing optical system, 54 ... Measurement Distance light receiving part, 55 .. Rotating mirror for high and low angle 56 ... Projection optical axis, 57 ... Condensing lens, 58 ... Image receiver, 59 ... Projection casing, 60 ... Flange, 61 ... Mirror holder plate, 62 ... Rotating shaft, 63 ... High / low angle gear, 64 ... High / low angle detector, 65 ... Drive motor for high / low angle, 66 ... Drive gear, 67 ... Terumoto Terumon, 68 ... External storage device, 69 ... Horizontal drive unit, 70 ... High / low drive unit, 71 ... Leveling drive unit, 72 ... Distance data processing unit, 73 ... Image data processing unit, 81 ... GPS antenna, 82 ... GPS receiving unit, 100 '... Optical information processing unit, 120 ... Building, 121 ... Passenger car, 122 ... Laser scanner, 123 ... Personal computer 124 ... Stereo pair image capturing device, 125 ... Digital camera, 126 ... Digital camera, 127 ... Mount.

Claims (10)

A three-dimensional point cloud position data receiving unit that receives the first three-dimensional point group location data according to the measurement object that is based on reflected light resulting laser beam in the first aspect,
A coordinate calculation unit for calculating second 3D point cloud position data relating to the measurement object based on a stereo image of the measurement object photographed at the second viewpoint;
A correspondence specifying unit that specifies a correspondence between the first three-dimensional point group position data and the second three-dimensional point group position data ;
An internal orientation element calculation unit that calculates an internal orientation element of an imaging apparatus that has captured the stereo image based on the first three-dimensional point cloud position data and the correspondence relationship specified by the correspondence relationship specification unit; ,
An image correcting unit that corrects the stereo image based on the internal orientation element calculated by the internal orientation element calculation unit ;
The first three-dimensional point group position data and the third three-dimensional point group position data related to the measurement object based on the stereo image corrected by the image correction unit are integrated, and the integrated three-dimensional An optical information processing apparatus comprising: a three-dimensional model creating unit that creates a three-dimensional model of the measurement object based on point cloud position data . The correspondence relationship specifying unit, based on the shape of the measurement object, the optical information processing apparatus according to claim 1, characterized in that to calculate the corresponding relationship. The correspondence relationship specifying unit, the intensity of the reflected light of said laser light, color, hue, performing one or a plurality selected from saturation, the specific of the correspondence based on the image of the measurement object The optical information processing apparatus according to claim 1 or 2.   The correspondence relation specifying unit specifies the correspondence relation based on a comparison between an image obtained by imaging the region irradiated with the laser light and an image taken by the image photographing unit. Item 3. The optical information processing apparatus according to Item 1 or 2. The coordinate calculation unit calculates the third 3D point cloud position data by calculating the 3D point cloud position data again based on the stereo image corrected by the image correction unit,
The correspondence specifying unit specifies the correspondence between the first three-dimensional point cloud position data and the third three-dimensional point cloud position data;
Based on the correspondence between the first three-dimensional point group position data and the third three-dimensional point group position data, the first three-dimensional point group position data and the third three-dimensional point group position 5. The optical information processing apparatus according to claim 1, wherein data integration is performed . The optical information processing apparatus according to claim 1, wherein the stereo image is obtained by a digital camera .   The image correcting unit corrects an image captured by the image capturing unit based on the focal length information of the image capturing unit obtained in advance or the internal orientation element calculated by the internal orientation element calculating unit in addition to the focal length information. The optical information processing apparatus according to claim 1, wherein the optical information processing apparatus is configured as described above. A first three-dimensional point cloud position data receiving means for receiving three-dimensional point group location data according to the measurement object that is based on the reflected light of the laser light obtained in the first aspect,
Coordinate calculating means for calculating second 3D point cloud position data relating to the measurement object based on a stereo image of the measurement object photographed at the second viewpoint;
Correspondence specifying means for specifying the correspondence between the first three-dimensional point group position data and the second three-dimensional point group position data ;
An internal orientation element calculating means for calculating an internal orientation element of an imaging apparatus that has captured the stereo image based on the first three-dimensional point cloud position data and the correspondence relation specified by the correspondence relation specifying means; ,
Image correcting means for correcting the stereo image based on the internal orientation element calculated by the internal orientation element calculation means ;
The first three-dimensional point group position data and the third three-dimensional point group position data related to the measurement object based on the stereo image corrected by the image correction unit are integrated, and the integrated three-dimensional An optical information processing system comprising: a three-dimensional model creating unit that creates a three-dimensional model of the measurement object based on point cloud position data . A first three-dimensional point group location data reception step of receiving three-dimensional point group location data according to the measurement object that is based on the reflected light of the laser light obtained in the first aspect,
A coordinate calculation step of calculating second 3D point cloud position data relating to the measurement object based on a stereo image of the measurement object photographed at the second viewpoint;
A correspondence specifying step for specifying a correspondence between the first three-dimensional point group position data and the second three-dimensional point group position data ;
An internal orientation element calculating step for calculating an internal orientation element of an imaging apparatus that has captured the stereo image , based on the first three-dimensional point cloud position data and the correspondence relationship specified in the correspondence relationship specifying step; ,
An image correction step for correcting the stereo image based on the internal orientation element calculated in the internal orientation element calculation step ;
The first three-dimensional point cloud position data and the third three-dimensional point cloud position data related to the measurement object based on the stereo image corrected in the image correction step are integrated, and the integrated three-dimensional An optical information processing method comprising: a three-dimensional model creation step of creating a three-dimensional model of the measurement object based on point cloud position data . A program that is read and executed by a computer,
Computer
A first three-dimensional point cloud position data receiving means for receiving three-dimensional point group location data according to the measurement object that is based on the reflected light of the laser light obtained in the first aspect,
Coordinate calculating means for calculating second 3D point cloud position data relating to the measurement object based on a stereo image of the measurement object photographed at the second viewpoint;
Correspondence specifying means for specifying the correspondence between the first three-dimensional point group position data and the second three-dimensional point group position data ;
An internal orientation element calculating means for calculating an internal orientation element of an imaging apparatus that has captured the stereo image based on the first three-dimensional point cloud position data and the correspondence relation specified by the correspondence relation specifying means; ,
Image correcting means for correcting the stereo image based on the internal orientation element calculated by the internal orientation element calculation means ;
The first three-dimensional point group position data and the third three-dimensional point group position data related to the measurement object based on the stereo image corrected by the image correction unit are integrated, and the integrated three-dimensional programs for optical information processing, characterized in that to function as a three-dimensional model creation means for creating a three-dimensional model of the object to be measured based on the point group location data.

Images