JP2011095858A - Three-dimensional digitizer - Google Patents

Abstract

Provided is a three-dimensional digitizer for generating three-dimensional information in a real image having continuity and generating image data capable of viewing an object included in the real image from an arbitrary viewpoint in a virtual space.
An image pickup means 2 picks up a color image, and a distance image generation means 1 has a field of view overlapping with the image pickup means 2 to generate a distance image. The imaging position measuring unit 4 measures an imaging position and an imaging direction by the imaging unit 2 and the distance image generating unit 1. The coordinate conversion unit 3 converts the device coordinates defined in the distance image into three-dimensional real space coordinates defined in the real space using the imaging position and imaging direction measured by the position measurement unit 4. By using the real space coordinates of the object obtained by the coordinate conversion means 3, the virtual space forming means 5 models the object to form a virtual space. Further, the appearance forming unit 6 performs mapping on the object arranged in the virtual space using the appearance of the object imaged by the imaging unit as a texture.
[Selection] Figure 1

Description

  The present invention relates to a three-dimensional digitizer that converts position information about an object such as a building into three-dimensional data.

  Conventionally, in order to create a 3D model of an urban area, a 3D city model is created by separating terrain and buildings using measurement data obtained by mounting a laser scanner that performs 3D measurement on an aircraft. A technique of automatically generating has been proposed (see, for example, Patent Document 1). Further, as a similar technique, in order to measure the three-dimensional coordinate value of the ground, a camera is mounted on the vehicle and a sensing unit that acquires camera position information is mounted on the vehicle, and the camera viewpoint measured by the sensing unit is measured. A technique for acquiring a three-dimensional shape of a road surface from a position and an image captured by a camera is also known (see, for example, Patent Document 2).

  Patent Document 1 describes that a three-dimensional urban space model is constructed by pasting textures on three-dimensional data of terrain and buildings and integrating them three-dimensionally. Further, in Patent Document 2, the shape of the slit surface is acquired from the road surface image, the slit surface is further divided into minute line segments, and the minute line segments are converted into orthographic projection coordinates as texture data. It is described that the texture data is mapped to the three-dimensional coordinate values of the road surface.

JP 2002-170102 A JP 2002-74323 A

  As described above, Patent Document 1 and Patent Document 2 describe performing texture mapping on three-dimensional data. However, in the technique described in Patent Document 1, there is a description that the top image is extracted from the aerial photograph data, but it is not clear how to obtain a texture to be pasted on the three-dimensional data. Further, in the technique described in Patent Document 2, an image of a road surface is divided into slit surfaces, and further, the slit surfaces are divided into minute line segments and used as texture data, and a minute texture is pasted in a tile shape. Therefore, there is a problem that it is difficult to configure a road surface image into a continuous image.

  The present invention has been made in view of the above-described reasons, and the object thereof is to enable a real image having continuity to have three-dimensional information, and an object included in the real image from an arbitrary viewpoint in the virtual space. It is an object of the present invention to provide a three-dimensional digitizer that generates image data that can be viewed.

  In order to achieve the above object, the present invention provides a distance image having a pixel value as a distance value between an imaging unit that captures a color image or a grayscale image, and an object that has a field of view overlapping with the imaging unit and exists in the field of view. A distance image generating means to be generated; an imaging position measuring means for measuring an imaging position and an imaging direction by the imaging means and the distance image generating means; and an imaging position and an imaging direction in which device coordinates defined in the distance image are measured by the position measuring means A virtual space is formed by modeling the object using the real space coordinates of the object obtained by the coordinate transformation means that converts to the three-dimensional real space coordinates specified in the real space using the coordinate transformation means A virtual space forming unit that performs mapping on the object arranged in the virtual space by using the appearance of the object imaged by the imaging unit as a texture. That.

  The virtual space forming unit evaluates the similarity of real space coordinates for a plurality of distance images in which at least one of the imaging position and the imaging direction measured by the imaging position measuring unit is different and the same object is included in the field of view. It is desirable to identify the same object and different objects for different distance images and connect the different distance images by overlapping the same objects.

  The virtual space forming means combines a plurality of distance images in which at least one of the imaging position and the imaging direction measured by the imaging position measuring means is different and the same object is included in the field of view, thereby combining a specific imaging position and imaging It is desirable to model an object that is in a blind spot in the direction.

  Furthermore, it is desirable that the virtual space forming unit performs modeling using a vector image obtained by simplifying the shape of the object into a geometric shape using the real space coordinates of the object.

  In this case, the display device includes a monitor device that displays an image of the virtual space formed by the appearance forming unit, and an operation device that performs an editing operation on the image of the virtual space displayed on the monitor device. When a reference shape is designated by the operation device in the virtual space image displayed on the monitor device, it is desirable to generate a geometric shape vector image using a relative relationship with the shape.

  Further, the virtual space forming means is a case where the same object modeled is included in the field of view of a plurality of distance images at different times, and the difference in the position of the object in the different distance images exceeds a specified range. If it is, it is desirable not to place the object in the virtual space.

  The virtual space forming means evaluates the similarity of objects included in the same position of the real space coordinates in a plurality of distance images with different dates and times, and different objects exist at the same position of the real space coordinates in different distance images based on the similarity. When it is determined that the object exists, it is desirable to place an object modeled from the latest distance image in the virtual space at the position.

  Further, the virtual space forming unit may associate the minimum distance between pixels constituting the virtual space with the position of the object in the virtual space.

  According to the configuration of the present invention, the field of view of the imaging unit and the distance image generation unit are overlapped, and the object is modeled in real space coordinates using the distance image, the imaging position, and the imaging direction, and modeling is performed. The object in the virtual space is mapped in the virtual space, and the appearance of the object imaged by the imaging means is mapped as the texture of the object in the virtual space, so the object in the virtual space has three-dimensional position information in the real space And has an appearance in real space. Therefore, it becomes possible to give three-dimensional information to a real image having continuity. That is, it is possible to generate image data that enables an object in real space to be viewed from an arbitrary viewpoint in the virtual space.

  In addition, for multiple distance images in which the same object is included in the field of view, the similarity of real space coordinates is evaluated, whether the objects included in each distance image are the same object or different objects, and the same object is superimposed In the configuration in which the distance images are connected together, it is possible to form a continuous virtual space without a break by using a plurality of distance images. Further, in the distance image, the closer the distance to the object, the higher the resolution. Therefore, by connecting the distance images that are close to the object, a high-definition virtual space can be constructed.

  If you adopt a configuration that models multiple objects that exist in the blind spot by combining multiple distance images that contain the same object in the field of view, the object that exists in the blind spot will be complemented with other distance images in any one of the distance images. Modeling can be performed.

  When using a configuration that uses a vector image in which the shape of the object is simplified to a geometric shape when modeling the object, the data required to describe the object compared to the case where the object is described using coordinate values for each part of the object The amount can be greatly reduced.

  Further, in the configuration in which an image of the virtual space is displayed on the monitor device and the shape serving as the reference position is indicated by the operation device, and the vector image of the geometric shape is generated using the relative relationship with the shape, for example, an object By specifying information such as the vertical plane, horizontal plane, or two planes that form a right angle, the amount of information for specifying the shape of the object increases by giving rules or knowledge about the shape of the object. Increases accuracy.

  When the same object included in distance images at different times has a difference in the position of the object that exceeds the specified range, the object is not placed in the virtual space. It is possible to form a virtual space in which an object that is excluded from the space and fixed at a fixed position is arranged. In this case, even if it is a person or a car, it cannot be removed from the virtual space if there is little movement, but by manually removing many of this kind of unnecessary objects from the virtual space, manual removal is not possible. Time and effort can be greatly reduced. In addition, since each object can be handled individually by modeling the object, it is easy to remove the individual object from the virtual space.

  When the object included in the same position of the real space coordinates for distant images with different date and time is dissimilar, the structure of the object modeled by the latest distance image is placed in the virtual space. The object in the virtual space can be automatically replaced with the latest information corresponding to the renovation. That is, since modeling is performed, an object such as a building can be handled individually, so that images can be easily replaced in units of objects.

  If the minimum distance between pixels constituting the virtual space is associated with the position of the object, the resolution can be changed according to the position of the object in the virtual space. For example, for objects that are inconvenient to show detailed images in the virtual space, set the minimum distance to a large value and reduce the resolution, and for objects that need to show detailed images, set the minimum distance to a low value and set the resolution. Can be raised.

It is a block diagram which shows embodiment. It is operation | movement explanatory drawing of the distance image generation means used for the same as the above. It is operation | movement explanatory drawing same as the above. It is operation | movement explanatory drawing same as the above. It is operation | movement explanatory drawing same as the above. It is operation | movement explanatory drawing same as the above.

  In the present embodiment, the distance image is generated by using the imaging unit 2 that captures a color image or a grayscale image, and the distance image generation unit 1 that generates a distance image using a distance value to an object existing in the field of view as a pixel value. And extracting the three-dimensional information of the object existing in the real space, and mapping the image captured by the imaging unit 2 to the three-dimensional information of the object in the virtual space associated with the real space. This makes it possible to form a virtual space having three-dimensional information while using an image of the photographed image. The imaging unit 2 may output either a color image or a grayscale image (monochrome image), but in the following description, a color image is used.

  An object of the present invention is to extract three-dimensional information related to an object existing at a fixed position such as a building mainly in an urban area. When imaging an object, a still image of a distance image and a color image may be sequentially generated while changing a shooting position and a shooting direction, and a moving image of a distance image and a color image may be generated. A moving image can be handled as a set of still images having substantially short time intervals.

  In the following description, a technique for generating a distance image for extracting three-dimensional information will be described first, and then a technique for forming a virtual space by combining a color image and a distance image will be described.

  To generate a distance image, signal light such as infrared light is projected from the light source to the visual field of the light receiving means, and reflected light from an object (including a person) existing in the visual field is received by the light receiving means. By using information corresponding to the time difference from light reception to light reception, the principle of time of flight is used to detect the distance to the object. That is, the active distance image generating means for projecting the signal light from the light source, receiving the space where the signal light is projected by the light receiving means, and detecting the distance to the object using the output of the light receiving means Is used.

  The light source emits signal light (modulated light) whose intensity changes over time, and calculates the phase difference between the signal light reflected by the object and received by the light receiving means and the projected signal light. It is used as information corresponding to the time difference until light reception. As the modulation waveform of the signal light, a sine wave, a triangular wave, a sawtooth wave, a square wave, or the like can be used. When a sine wave, a triangular wave, or a sawtooth wave is used, the period of the signal light is set to a constant period.

  In addition, when using a square wave, in addition to making the period of the signal light constant, the ratio between the ON period (light emitting source light emitting period) and the OFF period (light emitting source non-light emitting period) is changed randomly. It is also possible to adopt a technique for making them. Although the latter configuration will not be described in detail, the fact that the on period and the off period can be regarded as one-to-one due to randomness by using the on period and the off period many times is used. In this operation, the influence of ambient light having periodicity can be reduced.

  The light receiving means includes an imaging device in which a plurality of pixels are arranged two-dimensionally. As the image pickup element, a well-known configuration for picking up a gray image provided as a CCD image sensor or a CMOS image sensor can be used. When this type of image sensor is used, a wavelength selective filter and a neutral density filter are used and the intensity of signal light is increased so that saturation by ambient light can be prevented and signal light components can be detected. Further, it is desirable that the range for measuring the distance is a short distance (several meters).

  In order to reduce the influence of ambient light, it is desirable to use an image sensor that is specifically designed to have a structure suitable for the distance image generation means. As this type of imaging device, for example, a configuration is known in which the charge corresponding to the ambient light component is discarded and the charge corresponding to the signal light component is integrated to prevent saturation due to ambient light. By using this type of image sensor, it is possible to reduce components of ambient light or ambient light that are not signal light among the charges generated in the light receiving region, and it is possible to increase the dynamic range for the signal light component. .

  The image pickup element has a plurality of pixels (light receiving regions), and each light receiving region generates a charge having a charge amount corresponding to the light receiving intensity. Moreover, the light receiving sensitivity for each light receiving area can be controlled by an electric signal, and information including the phase difference between light projection and light reception is detected by controlling the timing of receiving the signal light as described below. Can do. In addition, in the image sensor, the charge generated in each light receiving region is accumulated over a period that is an integral multiple of the modulation period of the signal light (for example, 10000 periods) and then taken out to suppress the occurrence of abnormal values. ing.

  In the following, in order to help understanding, the following configuration is described as an example of a configuration example of the distance image generation unit, but this configuration is not intended to limit the present invention, but the modulation waveform of the signal light, the configuration of the imaging device, With respect to the control of the image sensor and the like, configurations provided in various known distance image generation means can be used.

  As shown in FIG. 1, the distance image generating means 1 used in the following description includes a light emitting source 11 that emits light (preferably using near infrared rays) and an image sensor 12 that receives light from a target space. Prepare. The light emitting source 11 is a light emitting element such as a light emitting diode or a laser diode that can obtain a light output proportional to the instantaneous value of the input. Further, in order to ensure the amount of light emitted from the light source 11, the light source 11 is configured using an appropriate number of light emitting elements.

  The light source 11 constitutes a light projecting unit together with a light projecting optical system 13 that projects the signal light output from the light source 11 to the target space. Further, the image sensor 12 constitutes a light receiving means together with a light receiving optical system 14 that makes light from the target space incident on the image sensor 12. The light projecting optical system 13 and the light receiving optical system 14 are arranged close to each other, and the distance between the light projecting optical system 13 and the light receiving optical system 14 can be substantially ignored with respect to the field of view. .

  The distance image generation means 1 includes a modulation signal generation unit 15 that outputs a modulation signal for driving the light emission source 11, and a light reception timing at the image sensor 12 based on the modulation signal output from the modulation signal generation unit 15. A timing control unit 16 that generates a prescribed light reception timing signal, and a distance image generation unit 17 that generates a distance image by obtaining a distance to an object existing in the target space using the light reception signal output from the image sensor 12. Provided.

  The modulation signal generation unit 15 generates a modulation signal whose output voltage changes with a sine waveform having a constant frequency (for example, 20 MHz), and supplies the modulation signal to the light emission source 11, as shown in FIGS. Signal light whose light output changes in a sine wave form is emitted from the light source 11. When a light-emitting diode is used as the light-emitting source 11, a signal voltage of a modulation signal is applied to the light-emitting diode through a current limiting resistor, thereby changing a current flowing through the light-emitting diode and emitting signal light.

  The image pickup device 12 can generate an electric charge according to the received light intensity only during a period synchronized with the received light timing signal by using an electronic shutter technique. In addition, the charge generated in the light receiving region is transferred to the light-shielded accumulation region and accumulated in the accumulation region in an accumulation period corresponding to a plurality of periods (for example, 10000 periods) of the modulation signal. Is taken out as a light receiving output.

  The timing control unit 16 generates a light reception timing signal synchronized with the modulation signal. Here, four different light receiving timing signals are defined for defining four different phases in one cycle of the modulation signal, and setting a light receiving period having a certain time width for each phase, and four kinds of light receiving timing signals for each accumulation period. 1 is provided to the image sensor 12.

  That is, the process of accumulating the charge generated in the light receiving region in the light receiving region defined by one type of light receiving timing signal in one accumulation period and taking out the accumulated charge as a light receiving output to the outside of the image sensor 12 is repeated four times. Light reception outputs corresponding to four types of light reception timing signals are taken out of the image sensor 12 in four accumulation periods.

Now, as shown in FIG. 2 (c), it is assumed that the light reception timing signal is defined by a phase different by 90 degrees in one cycle of the modulation signal. In this case, when the light reception outputs (charge amounts) corresponding to the respective light reception timing signals are A0, A1, A2, and A3, the phase difference ψ [rad] is expressed by the following equation.
ψ = (A0−A2) / (A1−A3)
If the frequency of the modulation signal is f [Hz], the time difference Δt from light projection to light reception is expressed as Δt = ψ / 2π · f using the phase difference ψ, so the speed of light is c [m / s]. Then, the distance to the object can be expressed as c · ψ / 4π · f.

  That is, the distance to the object can be obtained from the four types of light reception outputs (charge amounts) A0 to A3. The time width of the light receiving period can be set as appropriate so that an appropriate amount of received light can be obtained in the light receiving region (for example, a time width corresponding to a quarter period of the modulation signal can be used). ). However, the time width of each light receiving period needs to be equal to each other.

  In the distance image generation unit 17, in addition to the above-described processing for obtaining the phase difference ψ based on the light reception outputs (charge amounts) A0 to A3 and converting the phase difference ψ, processing described below can be performed. The distance image generation unit 17 is configured using a computer, and the above-described processing is realized by executing a program on the computer. Further, not only the distance image generation unit 17 but also the configuration excluding the light emission source 11 and the image sensor 12 is realized using a computer.

  In the above-described operation example, four types of light reception timing signals are used. However, the phase difference ψ can be obtained using three types of light reception timing signals, and two types of light reception timing signals can be obtained in an environment where there is no ambient light or ambient light. It is possible to obtain the phase difference ψ even with the light reception timing signal.

  In the above-described operation, since one light receiving region is used for one pixel, four accumulation periods are required to extract four types of light receiving outputs (charge amounts) A0 to A3 from the image sensor 12. However, if two light receiving regions are provided for one pixel, it is possible to generate electric charges corresponding to two types of light receiving timing signals in one cycle of the modulation signal. The received light output corresponding to the signal can be read out once. Similarly, if four light receiving regions are provided in one pixel, charges corresponding to four types of light receiving timing signals are generated in one cycle of the modulation signal, and light receiving outputs corresponding to the four types of light receiving timing signals are generated once. It becomes possible to read by.

  Since the above-described distance image generating means 1 uses the imaging element 12 in which a plurality of pixels are two-dimensionally arranged as a light receiving element for receiving light from the target space, the distance value is set as the pixel value of each pixel. As a result, a distance image is generated. The generated distance image is stored in the memory of the computer.

  Hereinafter, a technique for forming a virtual space using the distance image generated as described above together with the color image captured by the imaging unit 2 will be described.

  The distance image generating means 1 and the imaging means 2 are housed in a camera housing that enables hand-held imaging. By using a camera housing that can be held by hand, it is possible to capture not only the location along the roadway as in the case of in-vehicle use, but also the location where the terrain is complicated such as an alley formed in an urban area. become. Similarly, the inside of a building can be imaged.

  In addition, since it is possible to increase the resolution by capturing an image of a portion requiring detailed information from a short distance, the resolution of the image can be made variable.

  The imaging unit 2 is arranged so that the field of view overlaps with the light receiving unit in the distance image generating unit 1. Desirably, the visual field of the imaging means 2 is made to coincide with the visual field of the light receiving means. Specifically, as shown in FIG. 1, a half mirror M1 formed of a dichroic mirror is disposed between the image sensor 12 and the light receiving optical system 14, and in front of the light receiving surface of the image sensor 21 constituting the image pickup means 2. A configuration in which the mirror M2 is disposed is employed. The half mirror M1 is disposed so as to be inclined at an angle of 45 degrees with respect to the light receiving surface of the image sensor 12, and the mirror M2 is parallel to the half mirror M1 and inclined at an angle of 45 degrees with respect to the light receiving surface of the image sensor 21. Are arranged as follows.

  With this configuration, the half mirror M1 separates the infrared light incident on the image sensor 12 and the visible light incident on the image sensor 21, transmits the infrared light and reflects the visible light toward the mirror M2, and the mirror M2 is half The visible light from the mirror M1 is reflected so as to enter the image sensor 21. Therefore, the field of view of the distance image generating means 1 and the imaging means 2 can be matched by appropriately adjusting the distances between the imaging elements 12 and 21 and the light receiving optical system 14.

  Note that the above-described configuration is an example, and a configuration in which a part of the visual field of the distance image generation unit 1 and the imaging unit 2 overlap may be adopted in addition to a configuration in which the entire visual field overlaps. For example, a light receiving optical system corresponding to the image sensor 21 may be provided separately from the light receiving optical system 14 corresponding to the image sensor 12 constituting the distance image generating means 1. Alternatively, a configuration in which the image sensor 12 of the distance image generating unit 1 is also used as the image sensor 21 of the image capturing unit 10 can be employed. In this case, the operation for generating the distance image and the operation for capturing the color image are switched.

  In this way, it is possible to associate distance information with respect to an object in a color image with a simple adjustment by overlapping all or part of the field of view of the distance image generating unit 1 and the imaging unit 2 and matching the position of the field of view. become.

  In order to form a virtual space using the distance image and the color image, it is necessary to specify the position of the object in the real space. Since the relative position between the object existing in the real space and the distance image generating means 1 can be known from the distance image, if the apparatus coordinates are defined in the distance image generating means 1, the position of the object in the apparatus coordinates can be determined as the distance image. Can be obtained from

  Here, in the distance image, since the direction when the object is viewed through the center of the light receiving optical system 14 corresponds to the position of each pixel, and the pixel value is the distance value to the object, the position of the object is expressed in polar coordinates (spherical coordinates). It will be represented by. On the other hand, the position of an object in real space is easier to handle if it is represented by an orthogonal coordinate system using latitude, longitude, and height. Accordingly, the coordinate conversion means 3 is provided to convert the polar coordinate values obtained from the distance image output from the distance image generation means 1 into the coordinate values of the orthogonal coordinate system defined as the device coordinates.

  The coordinate conversion unit 3 includes a first conversion unit 3a that converts the coordinates of the object in the distance image from polar coordinates to device coordinates, and the three-dimensional real data that defines the device coordinates defined in the distance image generation unit 1 in real space. And a second conversion unit 3b for converting into spatial coordinates. For the coordinate conversion from the device coordinates to the real space coordinates in the second conversion unit 3b, not only the device coordinates but also information on the relative relationship of the device coordinates to the real space coordinates is required. Therefore, an imaging position measuring unit 4 that measures the imaging position and imaging direction of the distance image generating unit 1 is provided.

  The imaging position measuring means 4 is a high-precision GPS (global positioning system) that measures the imaging position in real space coordinates, a gyro sensor that measures the inclination of the distance image generating means 1 and the imaging means 2, the distance image generating means 1 and the imaging means. A geomagnetic sensor that detects the direction of 2 is combined. Here, high-accuracy GPS means a technique for setting the maximum error to about several tens of centimeters by combining information from an artificial satellite with ground station information. Therefore, the imaging position measuring unit 4 measures the three-dimensional coordinates of the imaging position in the real space coordinates and the imaging direction (the rotation angle around each three-dimensional coordinate axis) by the distance image generating unit 1 and the imaging unit 2. Can do. By combining the imaging position and the imaging direction in the real space coordinates obtained by the imaging position measuring unit 4 as described above with the coordinate position of the object in the device coordinates in the second conversion unit 3b of the coordinate conversion unit 3, The coordinate position of the object in space coordinates is calculated.

  Since the three-dimensional position information of the object is obtained from the calculated coordinate position of the object in the real space coordinates, the object modeling is performed so that the virtual space forming unit 5 can handle the object as a computer graphics object. I do. For the modeling, a well-known model such as a surface model or a wire frame model can be used. However, it is desirable to perform modeling for forming a vector image so as to reduce the amount of information.

  Here, it is determined using knowledge (rules) whether or not an object can be handled as a group of objects like a building, and for an object that can be handled as a group of objects, the object is defined as an individual object. To separate.

  For example, if there is knowledge that “if there is a gap between adjacent buildings, both buildings are different objects”, this knowledge can be used to separate both buildings as separate objects. . As knowledge for separating a building into individual objects, it is also possible to use information such as the interval between adjacent buildings and the shape and color of the building.

  When the object has a relatively simple shape as in the case of a building, the object is a vector image simplified to a geometric shape. Such a simplification is reasonable because many buildings in the city are rectangular parallelepiped, and the window frame is often rectangular.

  As described above, the coordinate position of the object is determined and the modeling of the object is performed, whereby the object model can be arranged in a virtual space constructed by computer graphics.

  By the way, in this embodiment, since an object such as a building in real space is arranged in association with the virtual space, the entire object cannot often be included in one distance image. In addition, the virtual space is required to be associated with a real space as wide as possible. Therefore, when the virtual space forming means 5 places an object in the virtual space, a series of continuous virtual spaces are formed by connecting a plurality of distance images.

  Specifically, as shown in FIG. 3, a plurality of (two in the illustrated example) distance images (and color images) P1 and P2 in which at least one of the imaging position and the imaging direction is different are used. In the illustrated example, it is assumed that the distance image P2 is acquired by shifting the imaging position after acquiring the distance image P1. The change in the imaging position is small compared to the field of view, and includes the same region of the object Ob in most of the field of view of both distance images P1 and P2. In the following description, the left-right direction in the figure is the x direction and the up-down direction is the y direction.

  In addition, it is necessary to consider the angle change (imaging direction) as well as the parallel movement (imaging position) for the movement of the visual field. In particular, since the distance image generating means 1 and the imaging means 2 are handheld, it is actually necessary to include the imaging direction, but here, only the parallel movement is considered in a simplified manner.

  As described above, when the two distance images P1 and P2 are obtained, first, the distance coefficients P1 and P2 are overlapped to calculate a correlation coefficient. That is, in this example, a correlation coefficient is used as the similarity. Thereafter, the process of obtaining the correlation coefficient by shifting the distance image P2 by one pixel in the x direction or the y direction with respect to the distance image P1 is repeated until the overlapping portion is not formed in both the distance images P1 and P2. The correlation coefficient is obtained, and the maximum value of the correlation coefficient is obtained. It is assumed that the overlapping state in which the correlation coefficient is the maximum represents the positional relationship between the two distance images P1 and P2, and a new distance image P2 is arranged at the position.

  Here, when calculating the correlation coefficient of both distance images P1 and P2, the correlation coefficient changes in accordance with the number of pixels in the overlapping portion, so normalization is performed by dividing by the number of pixels in the overlapping portion. In the above example, the distance image P2 is shifted by one pixel with respect to the distance image P1, but after defining a range in which the correlation coefficient is increased by shifting a predetermined plurality of pixels, the correlation coefficient is within that range. You may make it obtain | require the position where becomes the maximum. Alternatively, instead of adopting the procedure of displacing the distance image P2 in the x direction or y direction with respect to the distance image P1 and then displacing it in the other direction, the distance image P2 is spirally formed with respect to the distance image P1. It may be moved to. By adopting these methods, it is possible to shorten the time for obtaining the maximum value of the correlation coefficient.

  Furthermore, as will be described later, since the imaging position and the imaging direction are measured by the imaging position measuring means 4, if the range for calculating the correlation coefficient is narrowed down using this information, the maximum correlation coefficient is obtained. The processing load required to obtain the value is reduced, and the processing time is also shortened.

  When the position where the two distance images P1 and P2 are to be overlapped is determined as described above, a region that is not overlapped with the distance image P1 in the new distance image P2 (indicated by the hatched portion in FIG. 4) is virtually displayed. It is set as a region newly arranged in the space. By repeating such processing, the virtual space can be expanded.

  In the above operation example, the correlation coefficient of only the distance images P1 and P2 is obtained. However, the correlation coefficient may be obtained for the color image, and the position where the images are overlapped may be determined from both correlation coefficients. However, since the distance images P1 and P2 are not affected by external light and stable information is obtained as compared with the color image, sufficient accuracy can be obtained even if only the distance images P1 and P2 are used.

  If the virtual space is expanded by connecting the images as described above, there is a possibility that a part of the virtual space may be lost. Can be filled. In this case, since the images are connected according to the coordinate position, there is no problem even if there are overlapping portions in the captured spatial region. Further, if the area expanded by the new distance image is displayed on the screen of the monitor device 7 (described later), the missing area is added to the user so that the missing area is filled. Can be captured.

  In order to connect images, the following processing may be performed. Here, as shown in FIG. 5, there are a plurality of (two in the illustrated example) distance images P1 and P2 in which at least one of the imaging position and the imaging direction is different, and the same objects Ob1 and Ob2 are in the field of view. A plurality of distance images P1 and P2 included are used.

  Since it is necessary to determine whether or not the objects Ob1 and Ob2 included in these distance images P1 and P2 are the same, the objects Ob1 and Ob2 are modeled on the basis of the distance images P1 and P2, and the objects in real space coordinates After obtaining the coordinate positions of Ob1 and Ob2, the similarity is evaluated with respect to the coordinate positions of the objects Ob1 and Ob2, and when similarities that can determine the same objects Ob1 and Ob2 are obtained, different distance images P1 and P2 are obtained. The objects Ob1 and Ob2 are overlapped.

  For example, when the shapes of the objects Ob1 and Ob2 are represented using vector images, if the segment is a line segment, the degree of similarity can be evaluated by evaluating the distance between the coordinate positions of the end points, the distance between the end points (the length of the line segment), and the like. If it is a surface, the degree of similarity can be evaluated by evaluating the number of sides and vertices, the coordinate position of the vertices, the area, and the like.

  By repeating the above process, the distance images can be connected one after another, and a continuous virtual space can be formed. In the distance image, the closer the distance to the object, the higher the resolution. Therefore, a high-definition virtual space can be constructed by connecting the distance images that are close to the object.

  For example, it is desirable that the distance to the object is about 2 to 3 m. At such a distance, the depth of field is relatively large, and the amount of information contained in the field of view is relatively small, so that a highly accurate distance image is obtained. Is obtained. In other words, it is assumed that the user moves while taking an image. As means for movement, walking, bicycle, automobile, etc. may be selected as appropriate. Note that the above-described distance is a guide, and the distance to the object can be selected as appropriate. Further, when the distance to the object is small, the distance can be measured for the details of the object, and when the distance to the object is large, the distance can be measured for the overview of the object. Therefore, if it is inconvenient to generate a detailed distance image, the distance may be set so that an overview distance image is generated.

  In addition, since the distance measurement range by the distance image generation unit 1 is limited, information on the height direction of the distance image is limited. For example, 4 m is set as the limit distance in the height direction, and the height exceeds the limit distance. The distance value may not be used for a part exceeding the limit distance for the object. However, color images can be used regardless of the limit distance. Further, for a distant place exceeding the limit distance, the SN may be improved by capturing the same image a plurality of times and integrating the obtained color image.

  The above-described operation is a process of connecting distance images based on the same object included in different distance images, but in a specific distance image, an object that is a blind spot of another object and a distance value cannot be obtained. However, if there is an object that does not become a blind spot in another distance image in which at least one of the imaging position and the imaging direction is different, modeling is also performed for the object that is a blind spot in the specific distance image.

  In short, since coordinate positions in real space coordinates are obtained for all objects included in the distance image, even if the object is a blind spot in one of the distance images, the object is included in the other distance images. If it is, modeling of the object is performed using the coordinate position of the object. Furthermore, since it is often impossible to obtain complete information from a single distance image, it is desirable to complement an object that is in a blind spot using a plurality of distance images. In this case, by evaluating the similarity of the objects existing around the object, it is evaluated whether or not the object becomes a group of objects from a plurality of distance images, and is determined to be a group of objects. When possible, the object is placed in the virtual space.

  The virtual space formed by the above processing can be displayed on the screen of the monitor device 7 provided in the computer. In addition, since the virtual space has three-dimensional information, an editing operation such as designation or deletion of an object in the virtual space can be performed by using a keyboard or mouse provided in the computer as the operation device 8.

  The function of this kind of editing operation on a virtual space having 3D information is well known in graphics software that handles 3D graphics. However, in this embodiment, it is not necessary to perform an editing operation on the shape of a building. In the limit, it is only necessary to be able to perform editing operations such as designation of points, lines, surfaces and corners in the virtual space, and deletion of objects.

  When a vector image is used for modeling an object, a user (operator) designates a point, a line, a surface, or a corner by using the operation device 7 while viewing the image displayed on the monitor device 8. Thus, it is possible to indicate that these points, lines, surfaces, and corners are shapes serving as a reference of the object. For example, when a vertical plane, a horizontal plane, or two planes (corner portions) that form a right angle are designated by the operation device 7 and the characteristics of the designated part (vertical plane, horizontal plane, corner portion) are designated by the operation device 7, This makes it easier to specify the shape of the model, resulting in higher modeling accuracy. This technique is particularly effective when modeling an object with a vector image.

  Now, if the geometric shape of an object is simplified to a rectangular parallelepiped, it is possible to prevent the inclination of the surface due to image distortion by designating one surface to be a vertical surface. Also, by using the specified surface as a reference, a part of the information on the other surface is obtained using the relative relationship with the other surface, and the amount of information defining the other surface increases, so the other surface is also modeled Becomes easier. In particular, it is possible to temporarily set the three-dimensional information of the entire object by estimating the back surface not included in the distance image with the geometric shape. That is, it is possible to estimate three-dimensional information related to an object using knowledge (rules) such as a relative position with respect to another surface. In this case, the information about the back surface of the object that becomes a blind spot in the distance image is temporarily estimated information, but may be supplemented with other information if necessary.

  In principle, an object to be arranged in the virtual space is assumed to be an object such as a building existing at a fixed position, and it is desirable that a moving object such as a person or a car is not arranged in the virtual space. Therefore, the virtual space forming unit 5 evaluates the similarity of the object models generated from the plurality of distance images that are distance images having different time values at which the distance values are acquired and whose time difference is relatively short (up to about 10 minutes). When there is an object having a high degree of similarity and which can be regarded as substantially the same object, the difference between the positions of both objects in real space coordinates is obtained. If this difference exceeds the prescribed range, it is determined that the object is a moving object such as a person or a car and is automatically excluded from the virtual space.

  For example, as shown in FIG. 6, it is assumed that two distance images P1 and P2 having different time values are obtained, and objects Ob1, Ob3 and Oba that are included in both distance images P1 and P2 in common. Among them, the coordinate positions of the real space coordinates of the objects Ob1 and Ob3 are not substantially changed, and the coordinate position of the real space coordinates of the object Oba is changed in the direction of the arrow, so that the position of the object Oba And the object Oba is automatically excluded from the virtual space. In the illustrated example, the object Ob2 is not modeled because it is not included in the distance image.

  Here, even if it is a moving object, if the amount of movement is small, it cannot be removed from the virtual space, but since many of the moving objects are automatically removed, even if the rest are deleted manually by editing operation, Compared to the case where the moving object is manually removed, the amount of work is greatly reduced. Moreover, since it is possible to handle the object as an individual object by modeling the object, it is possible to delete each object individually by specifying each object as a unit, and individually from the virtual space. It is easy to remove the object.

  When the object is modeled by measuring the three-dimensional coordinates of the object existing in the real space as described above and a virtual space is formed by computer graphics, the three-dimensional coordinate position of the object is described in the virtual space. be able to. Here, in the present embodiment, since the color image whose field of view overlaps with the distance image is captured by the imaging unit 2, each correspondence in the virtual space can be obtained by using the correspondence relationship between the pixel positions of the distance image and the color image. It is possible to cut out a color image of a region corresponding to the surface of an object (object).

  The color image obtained by the imaging device 21 is input to the appearance forming unit 6, and the appearance forming unit 6 designates the coordinate position of the color image corresponding to the coordinate position of the object from the virtual space forming unit 5 and selects a part of the color image. The image is cut out and the cut out color image is mapped as the texture of the object arranged in the virtual space. That is, an image obtained as a color image is mapped on the surface of an object defined by a three-dimensional coordinate position in the virtual space. An object in the virtual space is modeled so as to have three-dimensional position information in the real space, and has an appearance in the real space by mapping.

  Further, as described above, since the limit distance in the height direction is set, only a color image having no distance information is used for a place higher than the limit distance. That is, a place higher than the limit distance maps a color image on a flat shape.

  The information on the three-dimensional position coordinates and texture generated in this way is stored in the database 9 and is displayed on the screen of the monitor device 8 by being read from the database 9 by the operation of the operation device 7 as necessary. It becomes possible.

  Since an object arranged in the virtual space has three-dimensional position information, a three-dimensional display device in which a lenticular sheet is arranged on the screen surface of the monitor device 8 is configured, and an image displayed on the monitor device 8 is three-dimensionally displayed. If data conversion of three-dimensional data for constructing a virtual space is performed so that an image (an image in which two images that differ by the amount of parallax when viewed through a lenticular sheet can be seen with the left and right eyes) is formed, a user (operator ) Makes it possible to visually recognize a stereoscopic image. Therefore, it is possible to display an image as if viewing a real space.

  In addition, since it is not a discrete image that is a combination of still images, it forms a seamless virtual space that is continuously connected, so the walk-through function can be used to specify temporal changes in the viewpoint position and field of view range. This makes it possible to provide a realistic display. If you use this walk-through function to display an image from your starting point to your destination, you can check the directions as if you were walking, making it easier to understand than a map. Directions are possible. In other words, even if you do not go to the site, you can walk through in the environment as if you were on the spot, so you can reach the destination without getting lost in the way you are.

  By the way, if information on the same place is acquired at appropriate time intervals (for example, units such as one month, one year, etc.), the information in the virtual space can be updated to the latest information. Here, it is desirable to acquire information periodically with a fixed time interval. As described above, since the virtual space can be seamlessly connected if the same object exists in the field of view regardless of the imaging position and imaging direction, the imaging position and imaging direction are completely in every imaging. There is no need to match.

  The virtual space forming unit 5 evaluates the similarity of objects included at the same position in real space coordinates for a plurality of distance images with different dates and times. Here, when the similarity is high, it is estimated that the same object exists, and the virtual space information is not changed. When the similarity is low, it is determined that an object different from the previous one exists at the coordinate position. To do. The similarity is given by a numerical value obtained by comparing the shape and dimensions of the object as described above. When it is determined that different objects exist at the same position in real space coordinates by using distance images with different dates and times, an object modeled from the latest distance image is placed in the virtual space. In addition, when the same object exists from distance images with different dates and times, the position accuracy can be improved by using the average of the positions of both objects.

  By the way, since objects are modeled in the virtual space and handled as a group of objects, it is easy to replace information in units of the object in the virtual space. The object can be automatically replaced with the latest information. Thus, by updating the virtual space information to the latest information for each building, the latest image is displayed on the monitor device 8, and the store image can be provided using the Internet or a recording medium. In this case, the advertising effect can be expected.

  As described above, since the object to be arranged in the virtual space is a vector image, it can be freely enlarged and reduced, and if the object is enlarged at an appropriate coordinate position in the virtual space, details of the place of interest A simple image can be displayed on the monitor device 8. In addition, when publishing an image using the Internet or a recording medium, since there are places where it is inconvenient to display in detail, the virtual space forming unit 5 determines the minimum distance between pixels constituting the virtual space. It is desirable to associate with the coordinate position of the object in space. That is, it is possible to limit the resolution for each object location.

  In the above-described configuration example, buildings in urban areas are mainly handled as objects. Therefore, when 3D city map information is used in combination, the accuracy of the coordinate positions of objects in the virtual space can be further increased. In this case, not only the objects that exist along the streets of urban areas are arranged in the virtual space, but also the inside of the building or the station premises is arranged in the virtual space, it is more convenient when used for the purpose of route guidance. Become.

DESCRIPTION OF SYMBOLS 1 Distance image generation means 2 Imaging means 3 Coordinate conversion means 4 Imaging position measurement means 5 Virtual space formation means 6 Appearance formation means 7 Operating device 8 Monitor apparatus 9 Database Ob1, Ob2, Ob3 Object Oba Object P1, P2 Distance image

Claims (8)

  An imaging unit that captures a color image or a grayscale image, a distance image generation unit that generates a distance image having a pixel value as a distance value to an object having a field of view overlapping with the imaging unit, and the imaging unit and the distance An imaging position measuring unit that measures an imaging position and an imaging direction by the image generation unit, and a three-dimensional rule that is defined in the real space using the imaging position and the imaging direction in which the device coordinates defined in the distance image are measured by the position measuring unit. A coordinate conversion means for converting to real space coordinates, a virtual space forming means for forming a virtual space by modeling an object using real space coordinates of the object obtained by the coordinate conversion means, and arranged in the virtual space A three-dimensional digitizer comprising: an appearance forming unit that performs mapping on an object by using an appearance of the object imaged by the imaging unit as a texture.   The virtual space forming unit evaluates the similarity of real space coordinates for a plurality of distance images in which at least one of the imaging position and the imaging direction measured by the imaging position measuring unit is different and the same object is included in the field of view. 3. The three-dimensional digitizer according to claim 1, wherein the same object and different objects are identified for different distance images, and the different distance images are connected by overlapping the same objects.   The virtual space forming means combines a plurality of distance images in which at least one of the imaging position and the imaging direction measured by the imaging position measuring means is different and the same object is included in the field of view, thereby combining a specific imaging position and imaging The three-dimensional digitizer according to claim 1 or 2, wherein modeling of an object existing in a blind spot in a direction is performed.   The said virtual space formation means performs modeling by the vector image which simplified the shape of the object to the geometric shape using the real space coordinate of the object, The any one of Claims 1-3 characterized by the above-mentioned. 3D digitizer.   A monitor device that displays an image of the virtual space formed by the appearance forming unit; and an operation device that performs an editing operation on the image of the virtual space displayed on the monitor device, and the virtual space forming unit includes: 5. A vector image of a geometric shape is generated using a relative relationship with respect to the shape when a reference shape is designated by the operation device in an image of a virtual space displayed on a monitor device. 3D digitizer.   The virtual space forming means is a case where the same object modeled is included in the field of view of a plurality of distance images at different times, and the difference in the position of the object in the different distance images exceeds a specified range. In this case, the object is not arranged in a virtual space. The three-dimensional digitizer according to any one of claims 1 to 5, wherein the object is not arranged in a virtual space.   The virtual space forming means evaluates the similarity of objects included in the same position of real space coordinates in a plurality of distance images with different dates and times, and different objects in the same position of real space coordinates in different distance images based on the similarity The three-dimensional digitizer according to any one of claims 1 to 6, wherein an object modeled from the latest distance image is placed in the virtual space when it is determined that the object exists. .   8. The three-dimensional image according to claim 1, wherein the virtual space forming unit associates a minimum distance between pixels constituting the virtual space with a position of an object in the virtual space. Digitizer.

Images