JP6116094B2 - Image search device - Google Patents

Description

  The present invention relates to an image search apparatus that searches for the same position in a stereo image, for example.

  In a stereo image, for example, in order to calculate a three-dimensional distance between two points, a technique is known in which a user manually designates a start point and an end point in two images (see Patent Document 1).

JP 2010-128068 A

  In the technique described in Patent Literature 1, two images are displayed by drawing an auxiliary line so that the user can suitably specify the start point and the end point. However, even if there is an auxiliary line, there is a possibility that a designated point designated in one image and a corresponding point designated corresponding to the designated point in another image may be shifted due to a human error or the like.

  The present invention has been made in view of the above-described circumstances, and provides an image search apparatus capable of suitably determining a corresponding point in another image corresponding to a specified point specified in one image. This is the issue.

In order to solve the above problems, an image search device according to the present invention includes first image data captured by a first imaging device and a second image captured by a second imaging device that is spaced a predetermined distance from the first imaging device. An image data storage unit that stores data and third image data captured by a third imaging device that is separated from the first imaging device by a predetermined distance, and outputs the first image data to a display device An image using an image data output unit, a designated point obtaining unit for obtaining a designated point in the first image data in accordance with an operation of the input device by a user, and an image using image data around the designated point in the first image data matched by, and a corresponding point search section for searching a corresponding point corresponding to the specified point in, respectively, respectively of the second image data and the third image data, said first The image capturing device, the second image capturing device, and the third image capturing device are arranged on a straight line and image the same direction, and the corresponding point search unit includes the second image data and the third image data. A search target area having a predetermined width is set in a direction orthogonal to the arrangement direction of the first imaging apparatus, the second imaging apparatus, and the third imaging apparatus, and the first area includes the designated point in the first image data In the search target area of the second image data and the third image data using the data, respectively, and when an area corresponding to the first area is found in the search target area, Using the second region data narrower than the first region including the designated point in the first image data, the regions corresponding to the first region of the second image data and the third image data are respectively searched. If no region corresponding to the first region is found in the search target region, the first region of the second image data and the third image data using the region obtained by dividing the first region An area corresponding to is searched .

In the image search device of the present invention, the first image data photographed by the first photographing device and the second image data photographed by the second photographing device separated by a predetermined distance from the first photographing device are associated with each other. And an image data output unit for outputting the first image data to a display device, and a designated point for acquiring a designated point in the first image data in accordance with an operation of the input device by a user. An acquisition unit; and a corresponding point search unit that searches for a corresponding point corresponding to the specified point in the second image data by image matching using image data around the specified point in the first image data , The first photographing device and the second photographing device are photographing in the same direction, and the corresponding point search unit is configured to use the first photographing device and the second photographing device in the second image data. A search target area having a predetermined width in a direction orthogonal to the arrangement direction of the apparatus is set, and the search target area of the second image data is set using data of the first area including the designated point in the first image data. When the area corresponding to the first area is found in the search target area, the data of the second area narrower than the first area including the designated point in the first image data When the area corresponding to the first area of the second image data is searched using and the area corresponding to the first area is not found in the search target area, the first area is divided. A region corresponding to the first region of the second image data is searched using the region obtained .

  According to the present invention, a corresponding point in another image corresponding to a designated point designated in one image can be suitably determined.

1 is a block diagram showing a panoramic image display system according to a first embodiment of the present invention. It is a schematic diagram for demonstrating arrangement | positioning of a board model. (A) is a figure for demonstrating the calculation principle of a 1st three-dimensional coordinate, (b) is an xz top view of (a). (A) is a figure for demonstrating the rotation around an X-axis, (b) is a figure for demonstrating the rotation around a Y-axis. It is a flowchart for demonstrating the operation example for determining the corresponding point corresponding to a designated point. (A) (b) is a schematic diagram for demonstrating the search method for determining the corresponding point corresponding to a designated point. (A) (b) is a schematic diagram for demonstrating the search method for determining the corresponding point corresponding to a designated point. It is a schematic diagram which shows the example which divided the 1st area into four. It is a block diagram which shows the panoramic image display system which concerns on 2nd embodiment of this invention. (A) is a figure for demonstrating the calculation principle of a 1st three-dimensional coordinate, (b) is an xz top view of (a). (A) (b) is a schematic diagram for demonstrating the search method for determining the corresponding point corresponding to a designated point. (A) (b) is a schematic diagram for demonstrating the search method for determining the corresponding point corresponding to a designated point.

  Hereinafter, an example in which the image search apparatus of the present invention is applied to panoramic image display will be described with reference to the drawings as appropriate. Similar parts are denoted by the same reference numerals, and redundant description is omitted.

<First embodiment>
As shown in FIG. 1, a panoramic image display system 1A according to the first embodiment of the present invention is a system that generates and displays an omnidirectional panoramic image inside a building, for example, and includes a compound eye photographing device 10A, An input device 20, a three-dimensional coordinate calculation device (image search device) 30, and a display device 40 are provided.

<Composite imaging device 10A>
The compound eye photographing device 10A includes a first monocular photographing unit (first photographing device) 11a, a second monocular photographing unit (second photographing device) 11b, and a third monocular photographing unit (third photographing device) 11c. The driving unit 12 for changing the shooting direction of the monocular imaging units 11a, 11b, and 11c, and the control unit 13 for driving and controlling the monocular imaging units 11a, 11b, and 11c and the driving unit 12 are provided.
The first monocular imaging unit 11a, the second monocular imaging unit 11b, and the third monocular imaging unit 11c are digital cameras, and the focal lengths of the second monocular imaging unit 11b and the third monocular imaging unit 11c are the same. f, the second monocular imaging unit from the left so that the optical axes of the imaging units 11a, 11b, and 11c are parallel to each other and the distance between the optical axes of the adjacent imaging units 11a to 11c is the distance d / 2. 11b, the first monocular imaging unit 11a, and the third monocular imaging unit 11c are arranged in this order, and are unitized while maintaining this arrangement. The focal length of the first monocular imaging unit 11a may be different from or the same as the focal length of the other monocular imaging units 11b and 11c.

The control unit 13 rotates the driving unit 12 to direct the monocular imaging units 11a, 11b, and 11c in a desired imaging direction, and controls the shutters of the monocular imaging units 11a, 11b, and 11c in this state to perform imaging. The captured image data relating to the captured image is output to the three-dimensional coordinate calculation device 30. In the present embodiment, the control unit 13 causes the monocular imaging units 11a, 11b, and 11c to perform imaging by rotating the compound eye imaging device 10 by a predetermined angle with the first monocular imaging unit 11a as the rotation center.
Further, the control unit 13 assigns an ID based on the shooting order (that is, the shooting direction) to the shot image data, and outputs the shot image data to which the ID is given to the three-dimensional coordinate calculation apparatus 30. Here, the 1st picked-up image data regarding the 1st picked-up image image | photographed by the 1st monocular imaging | photography part 11a, the 2nd picked-up image data regarding the 2nd picked-up image image | photographed by the 2nd monocular imaging | photography part 11b, 3rd Among the third photographed image data related to the third photographed image photographed by the monocular photographing unit 11c, those having the same photographing direction are assigned the same ID and associated with each other. Further, the second photographed image data and the third photographed image data correspond to the coordinate values in the X-axis and Y-axis directions among the coordinates of the first three-dimensional coordinate system described later in each photographed image.

<Input device 20>
The input device 20 includes a keyboard, a mouse, and the like. In response to a user operation, an input of image data designation instructions for a monocular panoramic image, designation point designations for the second and third photographed images, and the like to the three-dimensional coordinate calculation apparatus 30. Output.

<Three-dimensional coordinate calculation apparatus 30>
The three-dimensional coordinate calculation device 30 includes a CPU, a ROM, a RAM, an input / output circuit, and the like. As a functional unit, a captured image data storage unit 31, a map storage unit 32, a monocular panoramic image data generation unit 33, and the like. , Image data output unit 34, image data designation acquisition unit 35, designated point acquisition unit 36, first three-dimensional coordinate calculation unit 37, second three-dimensional coordinate calculation unit 38, and three-dimensional distance calculation unit 39. The three-dimensional coordinate calculation apparatus 30 is also an image search apparatus that searches for corresponding points corresponding to a specified point specified in one image data in other image data.

<< Photographed Image Data Storage Unit 31 >>
The captured image data storage unit 31 receives and stores a plurality of captured image data related to a plurality of captured images output from the control unit 13.

≪Map storage unit 32≫
The map storage unit 32 stores in advance a monocular panorama map used when generating a monocular panoramic image based on a plurality of first captured images (first captured image data) of the first monocular imaging unit 11a. .

≪Monocular panoramic image data generation unit 33≫
The monocular panoramic image data generation unit 33 reads a plurality of first captured image data output from the captured image data storage unit 31, and based on the read first captured image data, monocular panoramic image data related to the monocular panoramic image. Is generated.
Specifically, the monocular panoramic image data generation unit 33 reads a monocular panoramic map 50 (see FIG. 2) stored in advance in the map storage unit 32, and the first captured image captured by the first monocular imaging unit 11a. Monocular panorama image data is generated by mapping the data to the plate model 51 of the monocular panorama map 50.

  Here, the panorama map 50 will be described with reference to FIG. As shown in FIG. 2, the panoramic map 50 is configured by arranging a plurality of plate models 51 in a spherical shape. The center of the panorama map 50 (a sphere formed by a plurality of plate models 51) corresponds to the installation position of the imaging unit 11, and each plate model 51 is a first captured image by the first monocular imaging unit 11a. It corresponds to the shooting range. That is, the number of plate models 51 is the same as the number of first captured images by the first monocular imaging unit 11a, and the corners are formed by two line segments respectively connecting the center of the panorama map 50 and the centers of the adjacent plate models 51. Corresponds to the angle difference of the shooting direction by the first monocular imaging unit 11a when the adjacent first captured images are captured at the installation position of the first monocular imaging unit 11a. The aspect ratio corresponds to the aspect ratio of each first photographed image of the first monocular photographing unit 11a, and the distance from the center of the sphere formed by the plurality of plate models 51 to the plate model 51 and the aspect ratio of the plate model 51. The size corresponds to the focal length of the first monocular imaging unit 11a and the vertical and horizontal sizes of the captured image. Each plate model 51 is assigned an ID in the same manner as the first captured image data, and the monocular panoramic image data generation unit 33 is assigned to the ID and the plate model 51 assigned to the first captured image data. On the basis of the ID, the first captured image data is mapped to the corresponding plate model 51 to generate monocular panoramic image data, and the generated monocular panoramic image data is output to the image data output unit 34. Or stored in the storage unit 32. Here, the monocular panoramic image data generation unit 33 can generate monocular panoramic image data in a predetermined range centered on the viewpoint position based on the viewpoint position instruction output from the input unit 20.

<< Image Data Output Unit 34 >>
The image data output unit 34 outputs monocular panoramic image data to the display device 40. In the present embodiment, the monocular panoramic image data output from the monocular panoramic image data generation unit 33 is output to the display device 40. Or the second captured image data and the third captured image data read from the captured image data storage unit 31 are output to the display device 40.

<< Image Data Specification Acquisition Unit 35 >>
The image data designation obtaining unit 35 obtains designation of one first photographed image data from the monocular panoramic image displayed on the display device 40 in accordance with a user operation, and performs such designation result (for example, the designated first ID of the captured image data) is output to the image data output unit 34. Upon obtaining the designation result, the image data output unit 34 reads the first photographed image data corresponding to the designation result from the photographed image data storage unit 31, outputs the first photographed image data to the display device 40, and according to the designation result. The first captured image is displayed on the display device 40.

≪Specified point acquisition unit 36≫
The designated point acquisition unit 36 acquires a designated point instruction output from the input device 20 in the first captured image. Further, the designated point acquisition unit 35 uses the acquired designated point instruction to obtain two-dimensional coordinates (X-axis and Y-axis coordinate values of the designated point in the first captured image data stored in the captured image data storage unit 31. ) And is output to the corresponding point search unit 61.

≪Corresponding point search unit 61≫
The corresponding point search unit 61 searches for corresponding points corresponding to the designated point in each of the second photographed image data and the third photographed image data by matching using data around the designated point in the first photographed image data. Then, the two-dimensional coordinates of the found corresponding points are output to the first three-dimensional coordinate calculation unit 37. Here, the matching method by the corresponding point search unit 61 may be a method using the coincidence rate of the luminance distribution of the image data, and other well-known image matching methods can be applied.

≪First three-dimensional coordinate calculation unit 37≫
The first three-dimensional coordinate calculation unit 37 includes the two-dimensional coordinates of the corresponding points output from the corresponding point search unit 61, the distance d between the monocular imaging units 11b and 11c, and the focal length f of the monocular imaging units 11b and 11c. And the first three-dimensional coordinates of the corresponding points are calculated and output to the second three-dimensional coordinate calculation unit 38. Here, the first three-dimensional coordinate is a coordinate in the first three-dimensional coordinate system unique to each photographing direction with reference to the photographing direction of the monocular photographing units 11a to 11c, and the first three-dimensional coordinate system is the photographing. Also referred to as a three-dimensional coordinate system, the first three-dimensional coordinates are also referred to as photographing three-dimensional coordinates. That is, the first three-dimensional coordinate system is common to the second and third captured images having the same shooting direction, and is different between images having different shooting directions.

(First 3D coordinate calculation principle)
Here, the calculation principle of the first three-dimensional coordinate in the first three-dimensional coordinate calculation unit 37 will be described with reference to FIGS. In the following, the arrangement direction of the monocular imaging units 11a, 11b, and 11c is the X axis, the optical axis direction of the monocular imaging units 11a, 11b, and 11c is the Z axis, and the right hand coordinate system is composed of the Y axis that is orthogonal to the X axis and the Z axis. Consider a three-dimensional coordinate space. If the first monocular imaging unit 11a is arranged at the origin in the first three-dimensional coordinate system and the distance between the second monocular imaging unit 11b on the left side and the third monocular imaging unit 11c on the right side is d, the first three-dimensional coordinate O ₁ monocular imaging unit _11a, the first three-dimensional coordinates O ₃ of the first three-dimensional coordinate O _2, third monocular photography portion 11c of the second monocular photography section 11b, It becomes as follows.
O ₁ = (0, 0, 0)
O ₂ = (− d / _2, 0, 0)
O ₃ = (d / 2, 0, 0)

The object located at the relative three-dimensional coordinates R = (x _R , y _R , z _R ) is imaged using the monocular imaging units 11b and 11c, and the object is captured by the second monocular imaging unit 11b. The two-dimensional coordinates N ₂ = (x ₂ , y ₂ ), that is, the first three-dimensional coordinates (x ₂ , y ₂ , −f), and the third monocular imaging unit 11c captures the third captured image. It is assumed that the two-dimensional coordinates N ₃ = (x ₃ , y ₃ ), that is, the first three-dimensional coordinates (x ₃ , y ₃ , −f).
Here, since the triangle RO ₂ O ₃ and the triangle RN ₂ N ₃ are similar, the following relationship is established.
_{x R = d (x 2 +} x 3) / {2 (x 2 -x 3 + d)}
_{y R = d (y 2 +} y 3) / {2 (y 2 -y 3 + d)}
z _R = −df / {2 (x ₂ −x ₃ + d)}
That is, the first three-dimensional coordinate calculation unit 37 uses the predetermined focal distance f and interval d and the value of the two-dimensional coordinates of the photographed image to obtain the first three-dimensional coordinates of the desired position. R = (x _R , y _R , z _R ) can be calculated.

«Second three-dimensional coordinate calculation unit 38»
The second three-dimensional coordinate calculation unit 38 includes the first three-dimensional coordinates of the corresponding points output from the first three-dimensional coordinate calculation unit 37 and the shooting directions of the monocular shooting units 11b and 11c at the time of shooting. Based on this, the second three-dimensional coordinates of the corresponding point are calculated and output to the three-dimensional distance calculation unit 39 and the image data output unit 34. By doing so, the calculated second three-dimensional coordinates are displayed on the display device 40 described later. Here, the second three-dimensional coordinate is a coordinate in the second three-dimensional coordinate system that is common to all photographing directions, and is integrated with the first three-dimensional coordinate system, and the second three-dimensional coordinate system. Is also called a panoramic 3D coordinate system, and the second 3D coordinate is also called a panoramic 3D coordinate system. That is, the second three-dimensional coordinate system is common to all the second captured images and the third captured images.

(Second 3D coordinate calculation principle)
Here, the calculation principle of the second three-dimensional coordinate in the second three-dimensional coordinate calculation unit 38 will be described. The second three-dimensional coordinate calculation unit 38 sets the rotation angle around the Y axis from the reference of the shooting direction of the monocular imaging units 11b and 11c in the second three-dimensional coordinate system as θ, and the rotation angle around the X axis as φ. Then, the second three-dimensional coordinate A = (x _A , y _A , z _A ) of the designated point can be calculated by the following formula.

ここで、回転角度θ，φとしては、パノラママップ５０の板モデル５１に関連付けてマップ記憶部３２に予め記憶されたものを用いることができる。

Here, as the rotation angles θ and φ, those stored in advance in the map storage unit 32 in association with the plate model 51 of the panorama map 50 can be used.

Here, as shown in FIGS. 4A and 4B, the rotation of the angle φ around the X axis (X _R axis) of the relative three-dimensional coordinate and the rotation around the Y axis (Y _R axis) of the relative three-dimensional coordinate. by applying rotation and an angle theta, to the first three-dimensional coordinate system, X _R-axis relative three-dimensional coordinates, Y _R and Z axes (Z _R axis), X of the second three-dimensional coordinate system axis _{(X a-axis),} Y axis _{(Y a-axis)} and Z-axis and _{(Z a} axis) coincides respectively. Incidentally, phi, when viewed from the positive side of the X _A-axis counterclockwise positive, negative clockwise. θ, when viewed from the positive side of Y _A-axis counterclockwise positive, negative clockwise.
That is, the rotation around the Y axis (rotation in the pan direction) and the rotation around the X axis are converted into the first three-dimensional coordinates R = (x _R , y _R , z _R ) of the corresponding points in the first three-dimensional coordinate system. The second three-dimensional coordinates A = (x _A , y _A , z _A ) of the designated point are calculated by performing the rotation (rotation in the tilt direction). In the present embodiment, the origin in the second three-dimensional coordinate system and the origin in the first three-dimensional coordinate system, that is, O ₁ are the same position, and the rotation angle θ is equal to the Y-axis of the second three-dimensional coordinate system. The rotation angle φ is an angle formed by the X axis of the second three-dimensional coordinate system and the X axis of the first three-dimensional coordinate system. In the present embodiment, the X-axis in the relative three-dimensional coordinate system is always maintained in the horizontal direction, so that rotation conversion around the Z-axis is not necessary.

≪Three-dimensional distance calculation unit 39≫
The three-dimensional distance calculation unit 39 acquires the second three-dimensional coordinates of the two corresponding points output from the second three-dimensional coordinate calculation unit 38, and based on the acquired second three-dimensional coordinates, 2 A three-dimensional distance between two corresponding points is calculated and output to the image data output unit 34. Then, the image data output unit 34 can output the calculated three-dimensional distance to the display device 40 and display it on the display device 40.

<Display device 40>
The display device 40 displays a panoramic image using the panoramic image data output from the three-dimensional coordinate calculation device 30, or uses the second captured image data and the third captured image data to display the second captured image and the third captured image. Or display an image.

<Operation example>
Subsequently, for the operation example of the panoramic image display system 1A according to the first embodiment of the present invention, the second three-dimensional coordinates and the three-dimensional distances of designated points (start point and end point) included in different first captured images are obtained. An example of calculation will be described with reference to FIGS.

  First, the control unit 13 rotates the drive unit 12 to direct the monocular imaging units 11a, 11b, and 11c in a desired imaging direction, and controls the shutters of the monocular imaging units 11a, 11b, and 11c in this state. The photographed image data relating to the photographed image is output to the three-dimensional coordinate calculation device 30.

  Subsequently, the monocular panoramic image data generation unit 33 reads a plurality of first photographed image data photographed by the first monocular photographing unit 11a from the photographed image data storage unit 31, and uses the read first photographed image data as the read first photographed image data. Based on this, monocular panoramic image data is generated and output to the image data output unit 34. The image data output unit 34 outputs monocular panoramic image data to the display device 40, and the display device 40 displays a monocular panoramic image.

  In this state, when the user operates the input device 20 and designates the designated point P11 as the start point in the first photographed image 111 shown in FIG. 6A, the designated point instruction in the first photographed image 111 is the designated point acquisition. Input to the unit 36 (step S1 in FIG. 5).

  Subsequently, as illustrated in FIG. 6B, the corresponding point search unit 61 sets a first area (for example, 256 pixels × 256 pixels) A <b> 11 centered on the designated point P <b> 11 in the first captured image 111 ( Along with step S2), in the second captured image 112 and the third captured image 113 corresponding to the first captured image 111, a search target area A12 is set (step S3). Here, the search target area A12 is a band-like area, and is set over the entire arrangement direction of the monocular imaging units 11a to 11c in the second captured image 112 and the third captured image 113, and each monocular imaging unit 11a to 11a. 11c has a width that can include the first region A11 in a direction (vertical direction) orthogonal to the arrangement direction (lateral direction) of 11c. The corresponding point search unit 61 searches the search target area A12 of the second captured image 112 and the third captured image 113 by matching using the first area A11 (step S4).

  As shown in FIG. 7A, when a region A13 corresponding to the first region A11 is found in the search target region A12 (Yes in step S5), the corresponding point search unit 61 in the first captured image 111 A second region A14 (for example, 64 pixels × 64 pixels) narrower than the first region A11 including the designated point P11 as a center is set (No in step S6 and step S2), and the region A13 is set as a search target region. Set (step S3). The corresponding point search unit 61 searches the area A13 corresponding to the first area A11 of the second photographed image 112 and the third photographed image 113 by matching using the second area A14 (step S4).

  The corresponding point search unit 61 repeats the above operation by gradually reducing the size of the region centered on the designated point P11 in the first captured image 111 (256 × 256 → 64 × 64 → 8 × 8), and FIG. As shown in (b), a corresponding point P12 corresponding to the designated point P11 is determined (step S6).

  Here, for example, when the region A13 corresponding to the first region A1 is not found in the search target region A12 (No in step S5), as shown in FIG. The area A11 is divided into four parts, a lower right area A11a, a lower left area A11b, an upper left area A11c, and an upper right area A11d (step S11). By matching using the divided areas A11a to A11d, the second captured image 112 and the third The search area A12 of the photographed image 113 is searched (step S12).

Here, in the matching using the divided areas A11a to A11d, it is desirable to use the lower areas A11a and A11b before the upper areas A11c and A11d. In the present embodiment, matching is performed in the order of the lower right area A11a → the lower left area A11b → the upper left area A11c → the upper right area A11d, and the process proceeds to the next step when the corresponding area is found.
In actual captured images, there are many cases where a small object is placed on a large object, and the upper side is more visible than the lower side. Therefore, matching and accuracy can be achieved at the same time by using the lower regions A11a and A11b that are likely to have been photographed in the near side where the difference between the photographed images is small.

  When a region corresponding to any of the regions A11a to A11d is found in the search target region A12 (Yes in step S13), the corresponding point search unit 61 centers on the designated point P11 in the first captured image 111. A second area A14 (for example, 64 pixels × 64 pixels) narrower than the first area A11 to be included is set (No in step S6 and step S2), and other areas are complemented based on the discovered area. Thus, the area A13 corresponding to the first area A11 is set (step S3).

  In this operation example, steps S4 and S5 may be omitted, and the divided region setting in step S11 may be performed immediately after the search target region setting in step S3.

When the corresponding point search unit 61 determines the corresponding point P12 in the second captured image 112 and the third captured image 113, the two-dimensional coordinates of the corresponding point P12 in the second captured image 112 stored in the captured image data storage unit 31. (X ₁₁ , y ₁₁ ) and the two-dimensional coordinates (x ₂₁ , y ₂₁ ) of the corresponding point P 12 in the third captured image 113 are acquired and output to the first three-dimensional coordinate calculation unit 37.

Subsequently, the first three-dimensional coordinate calculation unit 37 calculates the first three-dimensional coordinates R _S = (x _RS , y _RS , z _RS ) of the corresponding point P12 as the start point based on the following formula, The data is output to the second three-dimensional coordinate calculation unit 38.
_{x RS = d (x 11 +} x 21) / {2 (x 11 -x 21 + d)}
_{y RS = d (y 11 +} y 21) / {2 (y 11 -y 21 + d)}
_{z RS = -df / {2 (} x 11 -x 21 + d)}

Subsequently, when the user operates the input device 20 and designates a designated point as an end point in the first photographed image, the corresponding point search unit 61 corresponds to the first photographed image by executing the flow of FIG. Corresponding points in the second photographed image and the third photographed image are determined, and the two-dimensional coordinates (x ₁₂ , y ₁₂ ) of the corresponding point in the second photographed image 122 and the third photographed image stored in the photographed image data storage unit 31. The two-dimensional coordinates (x ₂₂ , y ₂₂ ) of the corresponding points in the image 123 are acquired and output to the first three-dimensional coordinate calculation unit 37.

Subsequently, the first three-dimensional coordinate calculation unit 37 calculates the relative three-dimensional coordinates R _E = (x _RE , y _RE , z _RE ) of the corresponding point as the end point based on the following formula, Output to the three-dimensional coordinate calculation unit 38.
_{x RE = d (x 12 +} x 22) / {2 (x 12 -x 22 + d)}
y _RE = d (y ₁₂ + y ₂₂ ) / {2 (y ₁₂ −y ₂₂ + d)}
z _RE = −df / {2 (x ₁₂ −x ₂₂ + d)}

The second three-dimensional coordinate calculation unit 38 calculates the second and third second-order coordinates A _S = (x _AS , y _AS , z _AS ), A _E = (x _AE , y _AE ) based on the following formula. , Z _AE ) and outputs it to the three-dimensional distance calculation unit 39.

Subsequently, the three-dimensional distance calculation unit 39 calculates a three-dimensional distance D between the start point and the end point based on the following formula. Thereby, the three-dimensional distance is displayed on the display device 40.
^{_{D = {(x AE -x AS}} ) 2 + (y AE -y AS) 2 + (z AE -z AS) 2} 1/2

Since the panorama image display system 1A according to the first embodiment of the present invention searches the second photographed image and the third photographed image for the corresponding point corresponding to the designated point in the first photographed image, the user's manual operation Compared with the case where the corresponding point is determined by, the shift between the designated point and the corresponding point can be suppressed.
Further, the panoramic image display system 1A can calculate the second three-dimensional coordinates that can be used in common for all the images included in the panoramic image. Further, the panoramic image display system 1A can calculate a three-dimensional distance between two designated points. Further, the panoramic image display system 1 </ b> A has a second image corresponding to two captured images for acquiring specified points rather than the first captured image data corresponding to one captured image for a monocular panoramic image in one imaging direction. By setting the shooting range of each of the shot image data and the third shot image data wider, even if the vicinity of the periphery in the first shot image data is set as the specified point, the specified point is the second. Thus, the second three-dimensional coordinate and the three-dimensional distance can be reliably calculated because the captured image data and the third captured image data are surely within the photographing range.

<Second Embodiment>
Next, a panoramic image display system according to the second embodiment of the present invention will be described focusing on differences from the panoramic image display system 1A according to the first embodiment.

  As shown in FIG. 9, the panoramic image display system 1B according to the second embodiment of the present invention includes a compound eye photographing device 10B instead of the compound eye photographing device 10A. The compound eye photographing device 10B includes a first monocular photographing unit 11a and a second monocular photographing unit 11b having a focal length f, and the third monocular photographing unit 11c is omitted. The first photographed image data photographed by the first monocular photographing unit 11a in the present embodiment is used in the same manner as the first photographed image data and the second photographed image data in the first embodiment. The second captured image data captured by the second monocular imaging unit 11b is used in the same manner as the third captured image data in the first embodiment.

As shown in FIG. 8, the first monocular imaging unit 11a is arranged at the origin in the first three-dimensional coordinate system, and the distance between the left first monocular imaging unit 11a and the right second monocular imaging unit 11b. the When d, the first three-dimensional coordinate O ₁ of the first monocular photography portion _11a, the first three-dimensional coordinate O ₂ of the second monocular photography portion 11b is as follows.
O ₁ = (0, 0, 0)
O ₂ = (d, 0, 0)

An object located at relative three-dimensional coordinates R = (x _R , y _R , z _R ) is imaged using the monocular imaging units 11a and 11b, and the object is captured by the second monocular imaging unit 11a. The two-dimensional coordinates N ₁ = (x ₁ , y ₂ ), that is, the first three-dimensional coordinates (x ₁ , y ₁ , −f), and the second captured image of the second monocular imaging unit 11b. It is assumed that the _two -dimensional coordinates N ₂ = (x ₂ , y ₂ ), that is, the first three-dimensional coordinates (x ₂ , y ₂ , −f).
Here, since the triangle RO ₁ O ₂ and the triangle RN ₁ N ₂ are similar, the following relationship is established.
_{x R = dx 1 / (x} 1 -x 2)
_{y R = dy 1 / (x} 1 -x 2)
z _R = −df / (x ₁ −x ₂ )
That is, the first three-dimensional coordinate calculation unit 37 uses the predetermined focal distance f and interval d and the value of the two-dimensional coordinates of the photographed image to obtain the first three-dimensional coordinates of the desired position. R = (x _R , y _R , z _R ) can be calculated.

<Operation example>
Subsequently, regarding the operation example of the panoramic image display system 1B according to the second embodiment of the present invention, the second three-dimensional coordinates and the three-dimensional distances of the designated points S and E in the pillars included in the different first captured images are obtained. An example of calculation will be described with reference to FIGS.

  First, the control unit 13 rotates the drive unit 12 to direct the monocular imaging units 11a and 11b in a desired imaging direction, and in this state, controls the shutters of the monocular imaging units 11a and 11b to perform imaging and imaging. The captured image data relating to the image is output to the three-dimensional coordinate calculation device 30.

  Subsequently, the monocular panoramic image data generation unit 33 reads a plurality of first photographed image data photographed by the first monocular photographing unit 11a from the photographed image data storage unit 31, and uses the read first photographed image data as the read first photographed image data. Based on this, monocular panoramic image data is generated and output to the image data output unit 34. The image data output unit 34 outputs monocular panoramic image data to the display device 40, and the display device 40 displays a monocular panoramic image.

  In this state, when the user operates the input device 20 and designates the designated point P21 as the starting point in the first photographed image 121 shown in FIG. 11A, the designated point instruction in the first photographed image 121 is designated by the designated point acquisition unit. 36 (step S1 in FIG. 5).

  Subsequently, as illustrated in FIG. 11B, the corresponding point search unit 61 sets a first area (for example, 256 pixels × 256 pixels) A <b> 21 centered on the designated point P <b> 21 in the first captured image 111 ( Together with step S2), the search target area A22 is set in the second photographed image 122 corresponding to the first photographed image 121 (step S3). Here, the search target area A22 is set over the entire arrangement direction of the monocular imaging units 11a and 11b in the second captured image 112 and is orthogonal to the arrangement direction (lateral direction) of the monocular imaging units 11a and 11b. It has a width that can include the first region A21 in (vertical direction). The corresponding point search unit 61 searches the search target area A22 of the second captured image 122 by matching using the first area A21 (step S4).

  As shown in FIG. 12A, when a region A23 corresponding to the first region A21 is found in the search target region A22 (Yes in step S5), the corresponding point search unit 61 in the first captured image 121 A second area A24 (for example, 64 pixels × 64 pixels) narrower than the first area A21 including the designated point P21 as a center is set (No in step S6 and step S2), and the area A23 is set as a search target area. Set (step S3). The corresponding point search unit 61 searches the area A23 corresponding to the first area A21 of the second photographed image 122 by matching using the second area A24 (step S4).

  The corresponding point search unit 61 repeats the above operation by gradually reducing the size of the region centered on the designated point P21 in the first captured image 121 (256 × 256 → 64 × 64 → 8 × 8), and FIG. As shown in (b), a corresponding point P22 corresponding to the designated point P21 is determined (step S6).

  Here, for example, when the area A23 corresponding to the first area A21 is not found in the search target area A22 (No in step S5), the corresponding point search unit 61, as in the first embodiment, The one area A21 is divided into four parts, a lower right area, a lower left area, an upper left area, and an upper right area (step S11), and the search target area A22 of the second captured image 122 is searched by matching using the divided areas. (Step S12).

  When a region corresponding to any of the regions is found in the search target region A22 (Yes in step S13), the corresponding point search unit 61 includes the first including the designated point P21 in the first captured image 121 as the center. By setting a second area A24 (for example, 64 pixels × 64 pixels) narrower than the area A21 (No in step S6 and step S2), and complementing other areas based on the discovered area The area A23 corresponding to the first area A21 is set (step S3).

Corresponding point searching unit 61, the second determined the corresponding point P22 in the captured image 122, stored in the photographed image data storing section 31, the two-dimensional coordinates of the corresponding point P22 in the second captured image 122 _{(x 21, y 21} ) And is output to the first three-dimensional coordinate calculation unit 37 together with the two-dimensional coordinates (x ₁₁ , y ₁₁ ) of the designated point P21.

Subsequently, the first three-dimensional coordinate calculation unit 37 calculates the first three-dimensional coordinates R _S = (x _RS , y _RS , z _RS ) of the designated point P21 (corresponding point P22) as the start point based on the following formula. ) And output to the second three-dimensional coordinate calculation unit 38.
_{x RS = dx 11 / (x} 11 -x 21)
_{y RS = dy 11 / (x} 11 -x 21)
_{z RS = -df / (x 11} -x 21)

Subsequently, when the user operates the input device 20 and designates a designated point as an end point in the first photographed image, the corresponding point search unit 61 corresponds to the first photographed image by executing the flow of FIG. Corresponding points in the second photographed image are determined, two-dimensional coordinates (x ₂₂ , y ₂₂ ) of the corresponding points in the second photographed image stored in the photographed image data storage unit 31 are acquired, and the two-dimensional coordinates of the designated point P21 are obtained. The coordinates (x ₁₁ , y ₁₁ ) are output to the first three-dimensional coordinate calculation unit 37.

Subsequently, the first three-dimensional coordinate calculation unit 37 calculates the relative three-dimensional coordinates R _E = (x _RE , y _RE , z _RE ) of the designated point (corresponding point) as the end point based on the following formula. And output to the second three-dimensional coordinate calculation unit 38.
_{x RE = dx 12 / (x} 12 -x 22)
y _RE = dy ₁₂ / (x ₁₂ −x ₂₂ )
z _RE = −df / (x ₁₂ −x ₂₂ )

The second three-dimensional coordinate calculation unit 38, based on the following formula, designates and finishes the second three-dimensional coordinates A _S = (x _AS , y _AS , z _AS ), A _E = (x _AE , y _AE). , Z _AE ) and outputs it to the three-dimensional distance calculation unit 39.

Subsequently, the three-dimensional distance calculation unit 39 calculates a three-dimensional distance D between the start point and the end point based on the following formula. Thereby, the three-dimensional distance is displayed on the display device 40.
^{_{D = {(x AE -x AS}} ) 2 + (y AE -y AS) 2 + (z AE -z AS) 2} 1/2

  The panoramic image display system 1B according to the second embodiment of the present invention can omit one monocular photographing unit as compared with the panoramic image display system 1A.

  As mentioned above, although embodiment of this invention was described, this invention is not limited to the said embodiment, A design change is possible suitably in the range which does not deviate from the summary of this invention. For example, the number of area divisions in step S11 is not limited to four. Further, the method of notifying the user of the calculated second three-dimensional coordinates and the three-dimensional distance is not limited to display on the display device 40, and may be a configuration in which a speaker as a notification unit speaks. When the rotation around the Z axis (rotation angle ψ) is also taken into consideration, the second three-dimensional coordinate can be calculated by the following equation.

  In the present invention, the right-handed coordinate system is adopted as the first three-dimensional coordinate system and the second three-dimensional coordinate system, but a left-handed coordinate system can also be adopted.

1A, 1B Panoramic image display system 30 Three-dimensional coordinate calculation device (image search device)
31 Image data storage unit 33 Monocular panoramic image data generation unit 34 Image data output unit 35 Image data designation acquisition unit 36 Designated point acquisition unit 37 First three-dimensional coordinate calculation unit 38 Second three-dimensional coordinate calculation unit 39 Three-dimensional distance Calculation part 51 Corresponding point search part

Claims (2)

First image data photographed by the first photographing device, second image data photographed by the second photographing device separated from the first photographing device by a predetermined distance, and third image separated by a predetermined distance from the first photographing device. An image data storage unit that stores the third image data captured by the imaging device in association with each other;
An image data output unit for outputting the first image data to a display device;
A designated point obtaining unit for obtaining a designated point in the first image data in accordance with an operation of the input device by a user;
Corresponding points for searching corresponding points corresponding to the designated points in the second image data and the third image data by image matching using the image data around the designated points in the first image data. A search section;
Equipped with a,
The first photographing device, the second photographing device, and the third photographing device are arranged on a straight line and photograph the same direction,
The corresponding point search unit
In the second image data and the third image data, a search target area having a predetermined width is set in a direction orthogonal to the arrangement direction of the first imaging device, the second imaging device, and the third imaging device, and Search each of the search target areas of the second image data and the third image data using data of the first area including the designated point in the first image data,
When an area corresponding to the first area is found in the search target area, the second area data narrower than the first area including the designated point in the first image data is used. Search the areas corresponding to the first area of the second image data and the third image data,
If no region corresponding to the first region is found in the search target region, the first region of the second image data and the third image data using the region obtained by dividing the first region The image search apparatus according to claim 1, wherein an area corresponding to the image is searched. An image data storage unit that stores the first image data captured by the first image capturing device and the second image data captured by the second image capturing device spaced apart from the first image capturing device by a predetermined distance;
An image data output unit for outputting the first image data to a display device;
A designated point obtaining unit for obtaining a designated point in the first image data in accordance with an operation of the input device by a user;
A corresponding point search unit for searching for a corresponding point corresponding to the specified point in the second image data by image matching using image data around the specified point in the first image data;
Equipped with a,
The first photographing device and the second photographing device are photographing the same direction,
The corresponding point search unit
In the second image data, a search target area having a predetermined width is set in a direction orthogonal to the arrangement direction of the first imaging device and the second imaging device, and the first image data includes the designated point in the first image data. Search within the search target area of the second image data using area data,
When an area corresponding to the first area is found in the search target area, the second area data narrower than the first area including the designated point in the first image data is used. Search the area corresponding to the first area of the two image data,
If no area corresponding to the first area is found in the search target area, an area corresponding to the first area of the second image data is searched using an area obtained by dividing the first area. An image search apparatus characterized by:

Images