JP2002262157A - Multifocal omnidirectional imaging device - Google Patents

Abstract

(57) [Problem] To provide a multifocal omnidirectional imaging device capable of eliminating optical blur by changing a focus and reducing astigmatism in omnidirectional imaging using a mirror system. SOLUTION: In an apparatus for acquiring an omnidirectional image using a hyperboloid mirror 11, a light receiving lens 12, and an image sensor 13 that reflects an omnidirectional image, an acquired image range is divided into two or more, and the focus of the image sensor 13 is adjusted. Changing the position reduces astigmatism and obtains each optimal image focus.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an omnidirectional imaging device capable of obtaining an image of an omnidirectional panoramic landscape or the like.

[0002]

2. Description of the Related Art Conventionally, an omnidirectional vision sensor is an ultra-wide-angle sensor capable of capturing 360 ° omnidirectional images, and is used for surveillance cameras, video conferences and the like. In recent years, research and development for acquiring a panoramic image in a small size and in real time have been actively conducted as eyes of an autonomous mobile robot.

By the way, acquisition of visual information by a mobile robot or a monitoring device requires a small size and a wide visual range.

[0004] As a method of acquiring an omnidirectional panoramic image from a certain point in space in real time, there are a method of roughly capturing images from a plurality of cameras and a method of using a mirror or a fisheye lens.

The method using a plurality of cameras is, for example, a method of arranging an image sensor on each surface of a regular polyhedron and synthesizing the acquired images (for example, the prior art document: Omnidirectional camera system, 2nd "Regional area"). Symposium on Science and Technology Propagating from Japan, 1999.10.11). This method allows observation in all directions, but requires a large number of cameras, so that the apparatus becomes large and is not economical. In particular, there is a problem that it is too large in consideration of mounting on a robot.

On the other hand, a system using a mirror has been proposed by the present inventors and other prior applications [Patent: Patent No. 2
No. 939087 (Japanese Patent Application Laid-Open No. 6-295333), invention by the present inventors, Japanese Patent Application Laid-Open No. 6-141211, prior document: Yagi, Journal of the Robotics Society of Japan, Vol. 13, N
o. 3, p347-350, 1995, Sato, Yagi,
"Computer Vision: Technical Review and Future Prospects," New Technology Communications, 1998.

In these conventional systems, a hyperboloid-shaped mirror is arranged downward, a camera is arranged upward, and their vertical axes are aligned, so that the center of the camera lens is located at a position dual with the focal point of the hyperboloid. To Visual information is acquired by capturing an image reflected by the mirror with a camera. further,
The obtained image is subjected to a correction process and converted into an output image.
If this method is used, it is possible to image 360 degrees in all directions.

[0008]

However, the above-mentioned mirror system has a problem in improving the resolution because it is processed by one image sensor. The resolution is limited by the number of pixels of the image sensor and by optical blur. In contrast to the former, the inventors of the present application introduced a pixel shifting method into a system as a rotation method of an image sensor [Reference:
Nagahara et al., Information Processing Society of Japan, 2000-CVIM-121-
13 (2000.3.23)].

There are other proposals for improvement of the above-mentioned omnidirectional imaging of the mirror system. However, in any case, including the method of increasing the number of pixels, the resolution is insufficient, and at present it has not been put to practical use. In particular, no measures against optical blur have been taken.

In view of the above circumstances, the present invention provides a multifocal omnidirectional imaging apparatus capable of eliminating optical blur by changing focus and reducing astigmatism in omnidirectional imaging using a mirror system. The purpose is to:

[0011]

According to the present invention, there is provided an apparatus for acquiring an omnidirectional image by using a curved mirror for reflecting an omnidirectional image, a light receiving lens, and an imaging device. The acquired image range is divided into two or more parts, and astigmatism is reduced by changing the focal position of the image pickup device, so that the respective optimal image focuses are obtained.

[2] The multifocal omnidirectional imaging apparatus according to the above [1], further comprising means for changing a position of an imaging element with respect to the light receiving lens.

[3] The multifocal omnidirectional imaging device according to [2], wherein the focal position of the image sensor is moved in the vertical direction of the image sensor using a drive mechanism.

[4] The multifocal omnidirectional imaging device according to [2], wherein the focal position of the image sensor is moved in a horizontal direction using a driving mechanism.

[5] The multifocal omnidirectional imaging device according to [2], wherein the focus position of the image pickup device is moved in a rotational direction by using a drive mechanism.

[6] The multifocal omnidirectional imaging device according to [1], further comprising means for changing a position of a light receiving lens with respect to the imaging device.

[7] The multifocal omnidirectional imaging apparatus according to the above [4], further comprising means for changing a focus plane by electric control of the light receiving lens.

[8] The multifocal omnidirectional imaging apparatus according to the above [1], wherein a half mirror is provided after the light receiving lens, and a means for changing a focus plane is provided.

[0019]

[9] The multifocal omnidirectional imaging device according to the above [1], wherein a half mirror is provided in front of the light receiving lens, and a means for changing a focus plane is provided.

[10] The multifocal omnidirectional imaging device according to [1], wherein the curved mirror for reflecting the omnidirectional image, the light receiving lens, and the image sensor are rotated about a vertical line. It is.

[11] The multifocal omnidirectional imaging device according to [1], wherein the light receiving lens and the imaging device are rotated about a vertical line.

As described above, in an apparatus for acquiring an omnidirectional image using a hyperboloid mirror and an image sensor that reflects an omnidirectional image, the acquired image range can be divided into two or more, and an optimum image focus can be obtained for each. To do. The degradation of the optical resolution strongly depends on the depression angle of the incident light beam due to the effect of astigmatism.
On the other hand, the MTF limit when focusing on each depression angle is examined,
The design is made to exceed the limit resolution of the CCD over the entire depression angle range.

[0023]

Embodiments of the present invention will be described below in detail.

First, the optical system of the omnidirectional imaging device and the relationship between the input and the converted image will be outlined.

FIG. 1 is a diagram showing an optical system of an omnidirectional imaging device according to the present invention. FIG. 1 (a) is a schematic plan view thereof, and FIG. 1 (b) is a schematic side view thereof. FIG. 2 is a diagram showing the relationship between the input and the converted image, FIG. 3A is an omnidirectional input image,
FIG. 3B shows a converted image, and FIG. 3C shows an example of a panoramic image.

In these figures, 1 is a hyperboloid mirror, 2 is a focal point + c of the hyperboloid mirror 1, 3 is an incident light ray, 4 is an image plane (camera), 5 is a principal point -c of the lens, and 6 is a common perspective. The coordinates of the converted image, 7 are the coordinates of the panoramic image, and 8 are the coordinates of the input image.

As shown in these figures, the omnidirectional imaging device comprises a vertically downward hyperboloid mirror 1 and an upward camera 4. The hyperboloid mirror 1 is [(X ² −Y ² ) / a ² ] − (Z ² / b ² ) = − 1 (Z> 0) (1) c = √ (a ² + b ² ) (2) β = tan ⁻¹ (y / x) (3) α = tan ⁻¹ [(b ² + c ² ) sin γ−2bc] / (b ² + c ² ) cos γ (4) γ = tan ⁻¹ [f / {(X ² + y ² )]... (5), and the two focal points (0, 0, c), (0,
0, -c), the camera 4 is coaxial with the central axis of the hyperboloid, and the principal point 5 is one of the focal points (0, 0, -c) of the hyperboloid mirror 1.
It is arranged to come to. With this arrangement, it is possible to image 360 ° around the sensor at a time (see FIG. 3A). Further, as shown in FIG. 1, the light 3 entering toward the center of the mirror of the hyperboloid mirror 1 is reflected by the mirror due to the characteristics of the hyperboloid, and then goes to the other focal point. That is, the relationship between the arbitrary mapping point p (x, y) on the image, the depression angle α, and the azimuth angle β is uniquely determined from the above equations (3) to (5). From this equation, an omnidirectional input image [Fig.
2 (a)] can be easily converted to a perspective transformed image 6 [see FIG. 3 (b)] and a panoramic image 7 [see FIG. 3 (c)] from the viewpoint of the mirror as shown in FIG. Has characteristics.

As shown in the conventional example, since the omnidirectional imaging device can perform real-time conversion of an input omnidirectional image into a transparent conversion or a panoramic image without distortion, it presents a wide viewing angle image or an arbitrary viewpoint image to a person. It is suitable as an imaging sensor. However, as shown in FIG. 1, only one CCD
Since the sensor captures images in all directions, there is a disadvantage that the angle of view resolution is poor overall. Further, as shown in FIG.
Since the imaging target appears radially in the input image, the angle of view resolution decreases as the depression angle increases.

FIG. 4 is a diagram showing the angle-of-view resolution with respect to the depression angle and the optical blur characteristics.

This graph is a graph in which the angle of view resolution with respect to the depression angle in the peripheral direction and the radial direction of the input image is represented by a solid line, and the change of the optical blur characteristic with respect to the depression angle is represented by a dotted line.

As is apparent from this figure, when the panorama or perspective transformation is performed, the lower resolution of the lower part of the transformed image becomes more remarkable.

Under the same conditions, the MTF indicating the resolution is shown in FIG.
Shown in

FIG. 5 is a graph showing cutoff frequency characteristics of the MTF and the CCD with respect to the depression angle.

As can be seen from this figure, the resolution is controlled by the optical focus in the range where the depression angle is large and the range where the depression angle is small.

Next, the hyperboloid is divided into concentric circles, and the MTF for each depression angle is shown in FIG.

FIG. 6 is a diagram showing super-resolution processing using a multi-focus image sequence.

Here, 3 to -10 °, 20 ° and 40 °
I try to split it. In addition, 6 divisions (f _-10 f ₀ f
₁₀ f ₂₀ f ₃₀ f ₄₀ ).

As shown in this figure, the input and output image spaces are different. Therefore, the focus area (depth angle −20 to −2 °) of f _{−10 having} a small depression angle satisfies 90% of the output with approximately four input images, but the focus area of f ₄₀ (depth angles 21 to 21) having a large depression angle. 50 °), 50 sheets are required. For this reason, it is necessary to capture each image so that the input / output image ratio becomes uniform.

Then, the focus is adjusted for each division, and the obtained images are synthesized.

According to the present invention, in an apparatus for acquiring an omnidirectional image by using a hyperboloid mirror that reflects an omnidirectional image, a light receiving lens, and an image sensor, the acquired image range is divided into three or more, An object of the present invention is to provide a multi-focus omnidirectional imaging device that reduces astigmatism by changing a focal position of the image sensor and a unit that obtains image focus.

Hereinafter, embodiments of the present invention will be sequentially described.

FIG. 7 is a schematic diagram of a multifocal omnidirectional imaging apparatus showing a first embodiment of the present invention.

In this figure, 11 is a hyperboloid mirror, 12 is a light receiving lens, 13 is an image sensor, and 14 is a stepping gear as means for moving the image sensor 13 up and down.

In this embodiment, the position of the image sensor 13 with respect to the light receiving lens 12 can be changed by a driving mechanism, for example, a stepping gear 14.

According to this embodiment, since the image pickup device 13 is moved up and down, an effect of pixel shift is also produced. In the image, the resolution in the radial direction can be improved.

FIG. 8 is a schematic view of a multifocal omnidirectional imaging apparatus according to a second embodiment of the present invention.

In this embodiment, the position of the light receiving lens 12 with respect to the image pickup device 13 is changed. That is, the light receiving lens 12
Control device 15 as vertical moving means (drive mechanism)
Can change the focus plane.

In this embodiment, a commercially available camera can be used. Also, there is an advantage that a commercially available lens can be used.

FIG. 9 is a schematic view of a multifocal omnidirectional imaging apparatus according to a third embodiment of the present invention.

In this figure, 21 is a hyperboloid mirror, 22 is a light receiving lens, 23 is a half mirror arranged behind the light receiving lens 22, 24 is a first image pickup element arranged behind the half mirror 23, 25 Is a second imaging element arranged on the side of the half mirror 23.

With this configuration, the focus plane can be changed.

FIG. 10 is a schematic diagram of a multifocal omnidirectional imaging apparatus showing a fourth embodiment of the present invention.

In this figure, 31 is a hyperboloid mirror, 32 is a half mirror arranged before the first light receiving lens 33,
33 is a first light receiving lens arranged behind the half mirror 32, 34 is a first image sensor arranged behind the first light receiving lens 33, 35 is arranged on the side of the half mirror 32 The second light receiving lens 36 is a second image pickup device arranged on the side of the second light receiving lens 35.

With this configuration, the focus plane can be changed.

According to the third and fourth embodiments, there is an advantage that a multi-focus image can be captured in real time (operating speed of the image sensor) without a movable portion. It is also possible to sequentially combine the half mirrors to make the number of divisions of the image two or more. However, when the image is divided, the brightness of each image becomes dark. is there. Similar effects can be obtained with a prism other than the half mirror.

FIG. 11 is a schematic diagram of a multifocal omnidirectional imaging apparatus according to a fifth embodiment of the present invention.

In this figure, 41 is a hyperboloid mirror, 42 is a light receiving lens, and 43 is an image sensor. In this embodiment,
A movable part 45 is formed around a vertical line 44 passing through the hyperboloid mirror 41, the light receiving lens 42, and the image sensor 43.
5 is made to rotate the whole camera by the drive mechanism 46.

According to this embodiment, since the entire camera is rotated, there is an advantage that a conventional omnidirectional camera can be used as it is.

FIG. 12 is a schematic view of a multifocal omnidirectional imaging apparatus showing a sixth embodiment of the present invention.

In this embodiment, a vertical line 44 passing through the hyperboloid mirror 41, the light receiving lens 42 and the image pickup device 43 is used as a center.
The light receiving lens 42 and the imaging element 43 are formed as a movable part 47, and the movable part 47 is rotated by a driving mechanism 48.

With this configuration, the focus plane can be changed.

According to this embodiment, since the lens system is fixed, there is an advantage that the lens system is likely to be rotated from the external appearance, and it is difficult to cause a failure.

FIG. 13 is a schematic view of a multifocal omnidirectional imaging apparatus showing a seventh embodiment of the present invention. FIG. 13 (a) is an overall configuration diagram thereof, and FIG. 13 (b) is a parallel movement of the imaging device. FIG. 13C is a plan view of the displacement of the image sensor due to the rotation of the image sensor.

In this embodiment, only the image pickup device 43 is moved by the driving mechanism 49. That is, FIG.
As shown in FIG. 3B, the image sensor 43 is moved in parallel, or as shown in FIG.
Rotate 3

According to this embodiment, unlike the other pixel shifts, only the image sensor 43 is moved, so that there is an advantage that the movable part and the driving mechanism can be reduced in size. In addition, when the method is combined with the method of the first embodiment shown in FIG. 7, by moving only the image sensor 43, multiple focus and pixel shift can be performed, and the effect is large.

As described above, as an embodiment, a hyperboloid mirror and 4
Using a 100,000 single-chip CCD, three divisions (f _-Tenf_Tenf₄₀) And 6
Division (f_-Tenf₀f_Tenf₂₀f₃₀f₄₀) And each depression angle (right
The screen is divided according to the numbers below, and the image sensor
Reduce astigmatism by changing the focus position of the
You. The entire image is obtained by combining the images obtained by dividing
You. Along with the already proposed image shifting method,
Achieved practical image reproduction.

In the above embodiment, an omnidirectional camera using a hyperboloidal mirror has been described. However, the present invention is not limited to this. Other paraboloids, spherical surfaces, and two or more curved mirrors may be used. The omnidirectional camera used may be used.

In the above embodiment, the acquired image range is set to 3
Although the division is described above, the division of the acquired image range in the present invention can be applied if it is two or more.

It should be noted that the present invention is not limited to the above-described embodiment, but various modifications are possible based on the spirit of the present invention, and these are not excluded from the scope of the present invention.

[0070]

As described above, according to the present invention, the following effects can be obtained.

(A) In the omnidirectional imaging of the mirror system,
By changing the focus and reducing astigmatism, optical blur can be eliminated.

(B) By dividing the acquired image range into two or more parts and changing the focal position of the image pickup device, astigmatism can be reduced, and an optimum image focus can be obtained for each.

(C) In an autonomous robot, it is important to acquire omnidirectional image information in a small size, in real time, and with high resolution. According to the present invention, it is also useful for use as a fixed / mobile surveillance camera.

[Brief description of the drawings]

FIG. 1 is a diagram showing an optical system of an omnidirectional imaging device according to the present invention.

FIG. 2 is a diagram illustrating a relationship between an input and a converted image in an optical system of the omnidirectional imaging apparatus according to the present invention.

FIG. 3 is a diagram showing an omnidirectional input image and an example of a converted image of the omnidirectional imaging device according to the present invention.

FIG. 4 is a diagram showing an angle of view resolution and an optical blur characteristic with respect to a depression angle.

FIG. 5 is a diagram showing an MTF indicating resolution.

FIG. 6 divides a hyperboloid into concentric circles, and calculates M for each depression angle
It is a figure showing TF.

FIG. 7 is a schematic diagram of a multifocal omnidirectional imaging apparatus according to a first embodiment of the present invention.

FIG. 8 is a schematic view of a multifocal omnidirectional imaging apparatus according to a second embodiment of the present invention.

FIG. 9 is a schematic diagram of a multifocal omnidirectional imaging device showing a third embodiment of the present invention.

FIG. 10 is a schematic diagram of a multifocal omnidirectional imaging apparatus showing a fourth embodiment of the present invention.

FIG. 11 is a schematic view of a multifocal omnidirectional imaging apparatus according to a fifth embodiment of the present invention.

FIG. 12 is a schematic view of a multifocal omnidirectional imaging apparatus showing a sixth embodiment of the present invention.

FIG. 13 is a schematic diagram of a multifocal omnidirectional imaging apparatus showing a seventh embodiment of the present invention.

[Explanation of symbols]

 1,11,21,31,41 Hyperboloid mirror 2 Focus of hyperboloid mirror + c 3 Incident ray 4 Image plane (camera) 5 Main point of lens −c 6 Coordinate of common perspective transformed image 7 Coordinate of panoramic image 8 Input image Coordinates 12, 22, 42 Light receiving lens 13, 43 Image sensor 14 Up / down moving means (stepping gear) of image sensor 15 Moving means of light receiving lens (electric control device) 23, 32 Half mirror 24, 34 First image sensor 25 , 36 Second imaging element 33 First light receiving lens 35 Second light receiving lens 44 Vertical line 45, 47 Movable part 46, 48, 49 Drive mechanism

Continued on the front page (51) Int.Cl. ⁷ Identification symbol FI Theme coat II (reference) G03B 37/00 G06T 1/00 430H G06T 1/00 430 H04N 5/232 A E H04N 5/232 G03B 3/04 F term (Reference) 2H059 BA03 BA11 5B047 AA13 BB04 BB09 BC05 BC09 BC16 CA12 CA17 CB15 5C022 AA01 AB21 AB45 AB46 AB62 AB68 AC42 AC54 AC74

Claims (11)

[Claims] 1. An apparatus for acquiring an omnidirectional image using a curved mirror that reflects an omnidirectional image, a light receiving lens, and an image sensor, wherein an obtained image range is divided into two or more, and a focal position of the image sensor is changed. A multifocal omnidirectional imaging device that reduces astigmatism and obtains an optimal image focus for each. 2. The multi-focal omnidirectional imaging device according to claim 1, further comprising means for changing a position of an imaging device with respect to the light receiving lens. 3. The multi-focal omnidirectional imaging device according to claim 2, wherein the focus position of the image sensor is moved in a vertical direction of the image sensor using a driving mechanism. apparatus. 4. The multifocal omnidirectional imaging device according to claim 2, wherein the focal position of the image sensor is moved in a horizontal direction using a driving mechanism. 5. The multifocal omnidirectional imaging device according to claim 2, wherein the focal position of the image sensor is moved in a rotational direction using a driving mechanism. 6. The multifocal omnidirectional imaging device according to claim 1, further comprising: means for changing a position of a light receiving lens with respect to the image sensor. 7. The multi-focal omnidirectional imaging device according to claim 4, further comprising: means for changing a focus plane by electrically controlling the light receiving lens. 8. The multifocal omnidirectional imaging device according to claim 1, further comprising a half mirror provided after the light receiving lens, and a unit for changing a focus plane. 9. The multifocal omnidirectional imaging device according to claim 1, further comprising a half mirror disposed in front of the light receiving lens, and a means for changing a focus plane. 10. The multifocal omnidirectional imaging device according to claim 1, wherein the curved mirror that reflects the omnidirectional image, the light receiving lens, and the image sensor are rotated about a vertical line. . 11. The multifocal omnidirectional imaging device according to claim 1, wherein the light receiving lens and the imaging device are rotated about a vertical line.