JP2006309626A - Arbitrary viewpoint image generator - Google Patents

Abstract

PROBLEM TO BE SOLVED: To generate an arbitrary viewpoint image maintaining a feeling of depth only by image synthesis of simple processing.
An arbitrary viewpoint image generating apparatus according to the present invention divides a camera image obtained by photographing a photographing object using a camera into frame images, and a distance from the camera to the photographing object. The camera image storage unit 11 that stores information together with the information, the camera image space generation unit 13 that rearranges the stored frame images in a three-dimensional space based on the distance information, and the rearranged frame images. Thus, a viewpoint image generation unit 12 that generates a viewpoint image from a desired viewpoint position is provided.
[Selection] Figure 1

Description

  The present invention relates to an arbitrary viewpoint image generation device that generates an arbitrary viewpoint image using a camera image acquired by shooting an imaging target using a camera.

  Conventionally, as a method of generating an arbitrary viewpoint image based on a camera image acquired by shooting a real-world landscape, a three-dimensional model itself to be imaged is reconstructed from a plurality of images, and the three-dimensional model is converted into an arbitrary viewpoint. The observation method is known.

  In such a method, in order to model a subject to be photographed in a three-dimensional space, it is necessary to calibrate the camera with high accuracy before photographing, and it is difficult to reconstruct a three-dimensional model for an object having a complicated shape. There was a problem.

  In order to solve this problem, a method for generating and using an arbitrary viewpoint image without forming a three-dimensional model has been proposed.

  Such a technique is a technique called “image-based rendering”, and generally does not perform high-precision 3D shape restoration, but projects a captured image on a simple 3D model, thereby realizing a pseudo high-quality arbitrary viewpoint. A method for generating an image.

  Also, a method has been proposed in which a desired viewpoint image is cut out by recombining camera images into a panoramic shape without using a three-dimensional model to be photographed.

  On the other hand, the camera has been miniaturized, and not only an autofocus function using mechanical control but also a liquid lens that can realize a focus function by changing the lens thickness itself by electrical control has been proposed (Non-Patent Document). 1).

  When such a liquid lens is used, a mechanical part is not required, and by detecting the focal length of the lens, it is possible to easily measure distance information from the camera to the object to be photographed, and the lens can be greatly reduced in size. .

Further, as a device for directly measuring distance information, an ultra-small distance sensor using ultrasonic waves or the like has been developed.
Search the Internet, “www.varioptic.com/jp/technology.php”, April 28, 2005 Katada, Yamashita, Okuyama, Miyoshi, Kato, Aoyagi, Suzuki, "Array-type ultrasonic microsensor using PZT thin film" Physical Sensor Study Group, PHS-00-21, pp. 27-30, December 2000

  However, since the conventional methods as described above are intended to generate a high-quality arbitrary viewpoint image, it takes time to process, requires a plurality of cameras and a large apparatus, and real-time processing is difficult. In addition, there is a problem that it is difficult to downsize the apparatus.

  Therefore, the present invention has been made in view of the above points, and provides an arbitrary viewpoint image generation device that can generate an arbitrary viewpoint image that retains a sense of depth only by simple image synthesis. Objective.

  A first feature of the present invention is an arbitrary viewpoint image generation device, which divides a camera image obtained by photographing a photographing object using a camera into frame images, from the camera to the photographing object. A camera image storage unit that stores the frame information together with the distance information, a camera image space generation unit that rearranges the stored frame images in a three-dimensional space based on the distance information, and a composition of the rearranged frame images. The gist of the present invention is to provide a viewpoint image generation unit that generates an image from a desired viewpoint position or line-of-sight direction.

  In the first aspect of the present invention, the camera includes a distance measuring unit that measures the distance information, and the distance measuring unit is obtained from an autofocus function provided in the camera. The distance information may be measured based on the focal length.

  In the first aspect of the present invention, the camera may include a distance measuring unit that measures the distance information, and the distance measuring unit may measure the distance information using a distance sensor.

  In the first aspect of the present invention, a camera position determination unit that detects a moving direction of the camera and determines a position of the camera by performing image processing on the camera image is provided. The image space generation unit may rearrange each frame image in the three-dimensional space based on the determined position of the camera.

  In the first feature of the present invention, the camera includes a posture detection unit that detects the posture of the camera, and the camera image space generation unit three-dimensionally converts each frame image based on the detected posture of the camera. You may rearrange in the space.

  In the first aspect of the present invention, the camera image space generation unit may rotate and rearrange the frame image so that the line-of-sight direction of the designated camera is perpendicular to the frame image.

  In the first aspect of the present invention, the camera may include an image acquisition unit that captures the imaging target and a distance measurement unit that measures distance information from the camera to the imaging target.

  In the first feature of the present invention, the camera may have a camera shake correction function.

  As described above, according to the present invention, it is possible to provide an arbitrary viewpoint image generation device that can generate an arbitrary viewpoint image that retains a sense of depth only by simple image synthesis.

(Configuration of Arbitrary Viewpoint Image Generation Device According to First Embodiment of the Present Invention)
With reference to FIGS. 1 to 3, the configuration of the arbitrary viewpoint image generation apparatus according to the first embodiment of the present invention will be described.

  As shown in FIG. 1, an arbitrary viewpoint image generation apparatus 1 according to the present embodiment includes a camera 10, a camera image storage unit 11, a viewpoint image generation unit 12, a camera image space generation unit 13, and a viewpoint determination unit 14. It is equipped with.

  The camera 10 includes an image acquisition unit 101 that captures an imaging target and a distance measurement unit (focus adjustment unit) 102 that measures distance information from the camera 10 to the imaging target. The camera 10 also has a camera shake correction function.

  The image acquisition unit 101 is configured by a normal image acquisition device such as a CCD.

  Further, the camera 10 is equipped with an autofocus lens, and the autofocus lens corresponds to the distance measuring unit 102. In addition, such an autofocus lens can simultaneously detect the focal length of the camera 10 when focusing on a subject to be photographed.

  That is, the distance measuring unit 102 can measure distance information based on the focal length of the camera 10 acquired from the autofocus function provided in the camera 10.

  Further, as described above, the distance measuring unit 102 may measure the distance information using the autofocus function, or may measure the distance information separately from the camera image using a normal lens and a distance sensor. Good.

  As described above, the camera 10 can acquire the distance information from the camera 10 to the shooting target and the camera image of the shooting target simultaneously by shooting the shooting target.

  The camera image storage unit 11 is configured to divide a camera image acquired by shooting a shooting target using the camera 10 into frame images and sequentially store the frame image together with distance information from the camera 10 to the shooting target. Has been.

  The camera image space generation unit 13 is configured to rearrange each frame image stored in the camera image storage unit 11 in a three-dimensional space based on the distance information. Specifically, the camera image space generation unit 13 reads out each frame image from the camera image storage unit 11 and rearranges it in the photographing space (three-dimensional space).

  For example, as shown in FIG. 2, the shooting space 200 is a three-dimensional space (x, y, z), and the camera image space generation unit 13 includes a distance attached to each frame image in the shooting space 200. Each frame image is set up like a screen at a position corresponding to information.

For example, each frame image (210: i to 215: i + 5) of the camera image is given a value of “D _{i to} D _{i + 5} ” as distance information, and the camera image space generation unit 13 determines the distance. Each frame image is set up at the position of the z coordinate corresponding to the information.

  It is also conceivable that the camera image space generation unit 13 cuts out a shooting target by region division and repositions only the cut out shooting target.

  The camera image space generation unit 13 may be configured to rearrange the frame images in the three-dimensional space based on the posture of the camera 10 detected by the posture detection unit.

  Here, the camera image space generation unit 13 may be configured to rotate and rearrange the frame image so that the direction of the designated camera viewpoint is perpendicular to the frame image.

  As described above, since the posture of the camera 10 can be detected, in the example of FIG. 2, all the frame images (for example, 210 to 215) face the same direction, but the camera image space generation unit 13 The frame image can be rearranged in the three-dimensional space 200 so as to be perpendicular to the line of sight of the camera 10.

  The camera image space generation unit 13 may be configured to rearrange each frame image in the three-dimensional space based on the position of the camera 10 determined by the camera position determination unit.

  Here, the camera position determination unit may be configured to determine the position of the camera by detecting the moving direction of the camera 10 by performing image processing on the camera image.

  Here, the image processing indicates, for example, that the motion of the camera 10 itself is estimated by calculating an optical flow for each frame image and observing a change in the moving direction of the entire flow image.

  Further, the camera position determination unit may directly measure the tilt and moving direction of the camera 10 itself by using an acceleration sensor, a geomagnetic sensor, or the like attached to the camera 10 instead of the above-described image processing.

  In this case, as shown by 300 in FIG. 3, the frame images (301 to 305) are arranged in different directions, and when each frame image is observed from the viewpoint A as shown by 310 in FIG. 3 is obtained by combining frame images as indicated by 311 in FIG.

  Further, as shown by 320 in FIG. 3, when each frame image is observed from the viewpoint B, a frame image synthesis result as shown by 321 in FIG. 3 is obtained.

  The viewpoint determination unit 14 is configured to determine a desired viewpoint position, that is, a position where the camera 220 is arranged in the imaging space 200 based on an instruction from the user.

  The viewpoint image generation unit 12 is configured to generate an image from a desired viewpoint position (camera arrangement position) by combining the frame images 210 to 215 rearranged in the imaging space 200. .

  The viewpoint image generation unit 12 arranges the camera 220 at a position arbitrarily designated by the user in the three-dimensional space 200, cuts out an image viewed from the camera 220 from the frame image set up in a screen, The image is output as a viewpoint image at the camera position or an image in the viewing direction of the camera. For example, an image cut out by the camera 220 is configured by combining two frame images (212: i + 2, 213: i + 3) as indicated by reference numeral 221 in FIG.

(Operation of the arbitrary viewpoint image generation apparatus according to the first embodiment of the present invention)
With reference to FIG. 4, the operation of the arbitrary viewpoint image generation device according to the present embodiment will be described.

  As shown in FIG. 4, in step S <b> 101, the camera image storage unit 11 stores camera images (including distance information) acquired by the camera 10.

  In step S <b> 102, the camera image space generation unit 13 rearranges each frame image in the shooting space (three-dimensional space) 200 based on the above-described distance information.

  In step S103, the viewpoint image generation unit 12 arranges the camera at the viewpoint position determined by the viewpoint determination unit 14 in the shooting space 200, and in step S104, the viewpoint image or the viewpoint image or the image based on the image viewed from the arranged camera. A line-of-sight image is generated, and in step S105, the viewpoint image or the line-of-sight image is displayed.

  Here, with reference to FIG. 5, the example in the case of using the arbitrary viewpoint image production | generation apparatus which concerns on this embodiment for a small portable communication terminal is demonstrated.

  First, as illustrated in FIG. 5A, the user uses the camera-equipped mobile communication terminal 410 to photograph the photographing object 400. At this time, since the user moves by holding the camera-equipped mobile communication terminal 401 with his hand, the camera movement path 420 is not a straight line, but by performing camera shake correction, errors in alignment of each frame image are reduced. be able to.

  Here, the camera unit 410 in the camera-equipped mobile communication terminal 401 is configured by a camera equipped with a liquid lens. The camera unit 410 measures the distance information to the shooting target 400 shown in the center of the camera for each frame image while performing autofocus, and holds the distance information added to each frame image.

  Next, as shown in FIG. 5B, the display 412 in the camera-equipped mobile communication terminal 401 is based on the camera image stored in the memory of the terminal device 411 at the front viewpoint of the captured camera image. An image is displayed (see 431 in FIG. 5B).

  When the terminal device 411 main body is tilted by θ, the display 412 in the camera-equipped mobile communication terminal 401 generates and displays an image when the photographing object is observed from a viewpoint tilted by θ (FIG. 5B). 432).

  In this way, by freely tilting the terminal device 411 main body, the display 412 in the camera-equipped mobile communication terminal 401 can display an arbitrary viewpoint image. It can be recognized as an image of space.

(Operation / Effect of Arbitrary Viewpoint Image Generating Device According to First Embodiment of the Present Invention)
According to the arbitrary viewpoint image generation device according to the present embodiment, since the distance information is not included in the camera image itself acquired by the camera, the object to be photographed from the camera by the lens focus adjustment or the distance measurement device (sensor). The distance information is measured simultaneously, and the camera image is acquired in a form in which the distance information is added to each frame image. Then, by rearranging each frame image in the three-dimensional space according to the distance information, it is possible to reconstruct a three-dimensional space (scene) that holds depth information in a pseudo manner, although it is a writing effect. As a result, an arbitrary viewpoint image can be easily generated by arbitrarily arranging cameras in a three-dimensional space and constructing a desired camera image.

  In addition, according to the arbitrary viewpoint image generation device according to the present embodiment, what is arranged in the three-dimensional space is a flat frame image, but by rotating the frame image in the camera's viewpoint direction, it feels strange. It is also possible to relax.

  In addition, according to the arbitrary viewpoint image generation device according to the present embodiment, it is possible to easily reproduce an arbitrary viewpoint image from a camera image, and thus processing can be performed in real time. As a result, the user can complement and add multiple viewpoint images in the head by interactively specifying and observing arbitrary viewpoint images using the camera posture information. A sense of discomfort can be further eased, and an arbitrary viewpoint image can be observed naturally.

It is a functional block diagram of the arbitrary viewpoint image generation device concerning a 1st embodiment of the present invention. It is a figure for demonstrating a mode that each frame image is rearranged in three-dimensional space in the arbitrary viewpoint image generation apparatus which concerns on the 1st Embodiment of this invention. It is a figure for demonstrating a mode that a viewpoint image is produced | generated in the arbitrary viewpoint image production | generation apparatus which concerns on the 1st Embodiment of this invention. It is a flowchart which shows operation | movement of the arbitrary viewpoint image generation apparatus which concerns on the 1st Embodiment of this invention. It is a figure which shows a mode that the arbitrary viewpoint image generation apparatus which concerns on the 1st Embodiment of this invention is utilized with a small terminal.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 ... Arbitrary viewpoint image generation apparatus 10 ... Camera 101 ... Image acquisition part 102 ... Distance measurement part 11 ... Camera image storage part 12 ... Viewpoint image generation part 13 ... Camera image space generation part 14 ... Viewpoint determination part

Claims (8)

A camera image accumulating unit that divides a camera image obtained by photographing a photographing object using a camera into frame images and accumulates together with distance information from the camera to the photographing object;
A camera image space generation unit that rearranges each accumulated frame image in a three-dimensional space based on the distance information;
An arbitrary viewpoint image generation apparatus comprising: a viewpoint image generation unit that generates an image from a desired viewpoint position or line-of-sight direction by combining the rearranged frame images. The camera includes a distance measuring unit that measures the distance information,
The arbitrary viewpoint image generation apparatus according to claim 1, wherein the distance measurement unit measures the distance information based on a focal length of the camera acquired from an autofocus function provided in the camera. . The camera includes a distance measuring unit that measures the distance information,
The arbitrary distance image generation apparatus according to claim 1, wherein the distance measurement unit measures the distance information using a distance sensor. It comprises a camera position determination unit that determines the position of the camera by detecting the moving direction of the camera by performing image processing on the camera image,
The arbitrary viewpoint image generation apparatus according to claim 1, wherein the camera image space generation unit rearranges each frame image in a three-dimensional space based on the determined position of the camera. An attitude detection unit for detecting the attitude of the camera;
The arbitrary viewpoint image generation apparatus according to claim 1, wherein the camera image space generation unit rearranges each frame image in a three-dimensional space based on the detected posture of the camera.   2. The arbitrary viewpoint according to claim 1, wherein the camera image space generation unit rotates and rearranges the frame image so that the line-of-sight direction of the designated camera is perpendicular to the frame image. Image generation device. The camera
An image acquisition unit for photographing the photographing object;
The arbitrary viewpoint image generation apparatus according to claim 1, further comprising a distance measurement unit that measures distance information from the camera to the imaging target. The arbitrary viewpoint image generation apparatus according to claim 7, wherein the camera has a camera shake correction function.

Images