JP2011223284A - Pseudo-stereoscopic image generation device and camera - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide means for generating a pseudo-stereoscopic image signal which is free of a sense of incompatibility and close to a real image in any scene of a non-stereoscopic image when converting a non-stereoscopic image signal into a pseudo-stereoscopic image signal.SOLUTION: A CPU 108 calculates control signals CTL1, CTL2 and CTL3 for each frame based upon various kinds of input data such as the position of a range-finding area within a photographic image, an estimated distance to a subject, an estimated depth of field, data on whether a subject is a face, and data on the size of the face if the subject is a face, and supplies the data as parameters for pseudo-stereoscopic image generation to a 2D3D conversion section 115 which generates a pseudo-stereoscopic image from a non-stereoscopic image. The control signal CTL1 is used for controlling a composition ratio of individual image of a plurality of basic depth model types. The control signal CTL2 is a control signal indicating a weighting factor for weighting only an R signal component in a decoded image signal "a". The control signal CTL3 is a control signal that includes a parameter representing a depth and a parameter representing convergence.

Description

  The present invention relates to a pseudo-stereoscopic image generation apparatus and a camera, and more particularly to a pseudo-stereoscopic image from a normal moving image (non-stereoscopic moving image) in which depth information is not given explicitly or implicitly like a stereo image. The present invention relates to a pseudo three-dimensional moving image generation device and a camera that generate an image.

  In recent years, many stereoscopic image generation apparatuses that generate a stereoscopic image have been announced. This stereoscopic image generating apparatus can be roughly classified into three types. The first one includes a right-eye optical system and a left-eye optical system, and records and reproduces images captured by the respective optical systems. The second is a structure for measuring actual depth information in addition to the main optical system, and a two-dimensional still image or moving image (hereinafter referred to as a non-stereoscopic image) photographed by the main optical system. The three-dimensional image is constructed by combining the measured depth information.

  In the stereoscopic image generating apparatus disclosed in Patent Document 1, the distance is measured for each pixel of the screen of a two-dimensional image that is a non-stereo image. Then, the distance between all the subjects in the screen and the camera is measured using two cameras or laser beams, and distance information about each pixel of the screen of the two-dimensional image is obtained. Subsequently, in this stereoscopic image generating apparatus, in order to easily generate a depth image and a depth accuracy image reflecting the intention of the photographer at the time of shooting, the distance is determined by a non-linear function (mapping table) for each individual pixel. The depth value is calculated from the information. At this time, a plurality of mapping tables are selected according to the shooting information. Further, in this stereoscopic image generating device, with respect to the accuracy of the depth information, a portion with high accuracy and a portion with low accuracy can be provided in the screen.

  The third one is a stereoscopic image generating device that generates pseudo depth information by estimating from a non-stereo image, and composes the stereoscopic image by synthesizing the pseudo depth information and the non-stereo image. Since this stereoscopic image generation device generates pseudo depth information by estimation, it does not measure the distance for each pixel of the screen of the two-dimensional image.

  Further, in Patent Document 2 by the present applicant, when generating a pseudo-stereoscopic image, depth information estimated by an automatic processing method is used as a scene for each screen of a non-stereoscopic image (hereinafter referred to as a scene). A pseudo-stereoscopic image generation device that can be corrected accordingly is disclosed.

  In the pseudo stereoscopic image generating apparatus disclosed in Patent Document 2, a plurality of images having a sense of depth (hereinafter referred to as basic depth model type) are prepared, and a high frequency component of a luminance signal in one screen of a non-stereo image is calculated. Then, the composition ratio of the plurality of basic depth model types is automatically calculated based on the calculated value. Then, depth data for producing a sense of depth of the non-stereo image is estimated from the calculated composition ratio, and a pseudo-stereo image is generated from the non-stereo image and the depth data.

  At this time, if the composition ratio of a plurality of basic depth model types is automatically calculated by the same method predetermined for all non-stereo images, appropriate depth information may not be obtained depending on the scene, and it may be uncomfortable. A certain pseudo-stereoscopic image may be generated. However, it is practically difficult for the user himself to adjust the algorithm and parameters of the pseudo stereoscopic image each time according to the scene of the non-stereoscopic image. Therefore, in the pseudo-stereoscopic image generation device described in Patent Document 2, in order to generate a pseudo-stereoscopic image that is closer to the actual image without any sense of incongruity, the producer side corrects the depth information according to the scene. It is aimed.

JP 2008-141666 A JP 2009-044722 A

  However, the stereoscopic image generating apparatus described in Patent Document 1 requires two cameras or distance measuring means using laser light to acquire distance information, which makes the apparatus expensive. Further, in this stereoscopic image generating apparatus, when recording photographed information, it is necessary to record depth information for each pixel together with two-dimensional image data, and the recorded image data becomes large and can be stored and transmitted. Such costs are also a problem.

  On the other hand, Patent Document 2 does not disclose a specific method for adjusting depth information. For example, the person may actually watch the scene and manually determine the adjustment parameter. However, when converting a non-stereoscopic image captured by a user with a handy video camera to a pseudo-stereoscopic image, the pseudo-stereoscopic image generation apparatus described in Patent Document 2 is adjusted manually by the user as a producer. Although the parameters are adjusted, the adjustment for generating a pseudo-stereoscopic image closer to the actual image without a sense of incongruity is very troublesome.

  The present invention has been made in view of the above points, and when converting a non-stereoscopic image signal obtained by a video camera into a pseudo-stereoscopic image signal, any non-stereoscopic image scene can be obtained. It is an object of the present invention to provide a pseudo-stereoscopic image generation apparatus and a camera that can generate a pseudo-stereoscopic image signal closer to an actual image without a sense of incongruity.

  In order to achieve the above object, the pseudo-stereoscopic image generation apparatus of the present invention photoelectrically converts an optical image of a subject formed on an imaging surface of an imaging element through an optical system including a zoom lens, a focus lens, and a diaphragm by the imaging element. When the subject is a person from the non-stereo image signal obtained by the conversion, a face size information acquisition unit that acquires face size information of the person and a pseudo stereo image signal from the non-stereo image signal are generated. A basic depth model generating means for generating a plurality of basic depth model type images indicating a basic scene, and a basic depth based on a first control signal indicating a composition ratio of the plurality of basic depth model type images. A depth model synthesizing means for synthesizing a plurality of basic depth model type images supplied from the model generating means to generate a depth model image; and a non-stereoscopic image signal Weighting means for multiplying a non-stereo image signal by a weighting coefficient indicated by the second control signal based on a second control signal indicating a weighting coefficient for weighting, and the non-stereo image weighted by the weighting means The depth and the congestion are adjusted by the depth estimation data generating means for generating the depth estimation data from the signal and the depth model image generated by the depth model synthesizing means and the third control signal indicating the depth and the congestion. Based on the estimated depth estimation data, the texture shifting means for generating the pseudo stereoscopic image signal by shifting the texture of the non-stereo image signal and the first to third control signals are calculated and output. Control signal calculation that varies the value of at least one of the first to third control signals to be calculated using the magnitude information of And having a stage.

  In order to achieve the above object, the pseudo-stereoscopic image generation apparatus of the present invention is acquired by a lens position acquisition unit that acquires each position of a zoom lens and a focus lens that constitute an optical system, and a lens position acquisition unit. Subject estimated distance calculating means for calculating an estimated distance from the focus lens position to the subject based on the zoom lens position and the focus lens position, and an optical image of the subject imaged on the imaging surface of the image sensor through the optical system A basic depth model generating means for generating a plurality of basic depth model type images indicating a basic scene for generating a pseudo-stereoscopic image signal from a non-stereoscopic image signal obtained by photoelectric conversion by an imaging device; Based on the first control signal indicating the composition ratio of the images of the plurality of basic depth model types, the basic depth model generating means Depth model synthesis means for synthesizing a plurality of supplied basic depth model type images to generate a depth model image, and a second control signal indicating a weighting coefficient for weighting the non-stereo image signal The weighting means for multiplying the non-stereo image signal by the weighting coefficient indicated by the second control signal, the non-stereo image signal weighted by the weighting means, and the depth model image generated by the depth model synthesis means The texture of the non-stereoscopic image signal is shifted based on the depth estimation data generating means for generating the depth estimation data and the depth estimation data in which the depth and the convergence are adjusted by the third control signal indicating the depth and the congestion. The texture shift means for generating the pseudo stereoscopic image signal and the first control signal and the third control signal are respectively calculated and output. And a control signal calculating means for varying the value of at least one of the first and third control signals to be calculated using the information on the estimated distance to the subject. Features.

  In order to achieve the above object, the pseudo-stereoscopic image generation apparatus of the present invention includes an optical system information acquisition unit that acquires a position of a zoom lens, a position of a focus lens, and an aperture value of a diaphragm that constitute an optical system, Based on the aperture value and zoom lens position acquired by the optical system information acquisition means, the depth of field calculation means for calculating the estimated depth of field, and the subject imaged on the imaging surface of the image sensor through the optical system Basic depth model generation means for generating a plurality of basic depth model type images indicating a basic scene for generating a pseudo-stereoscopic image signal from a non-stereoscopic image signal obtained by photoelectrically converting an optical image by an image sensor. And a plurality of basic depth models supplied from the basic depth model generating means based on a first control signal indicating a composition ratio of the images of the plurality of basic depth model types. A non-stereo image signal based on a second model control signal indicating a weighting coefficient for weighting the non-stereo image signal, and a depth model synthesizing unit that synthesizes the image of the image and generates a depth model image Depth for generating depth estimation data from weighting means for multiplying the weighting coefficient indicated by the second control signal, the non-stereo image signal weighted by the weighting means, and the image of the depth model generated by the depth model synthesis means. Based on the depth estimation data in which the depth and the convergence are adjusted by the estimated data generation means and the third control signal indicating the depth and the congestion, the texture of the non-stereo image signal is shifted to generate a pseudo stereoscopic image signal. Information of the estimated depth of field calculated when calculating and outputting the texture shift means and the first to third control signals respectively Used, and having a variable control signal calculating means a value of at least one of the control signals of the first to third control signals to be calculated.

  In order to achieve the above object, the pseudo-stereoscopic image generation device of the present invention includes a lens position acquisition unit that acquires each position of a zoom lens and a focus lens that constitute an optical system, and imaging of an image sensor through the optical system. Of the non-stereo image signal obtained by photoelectrically converting the optical image of the subject imaged on the surface by the image sensor, the non-stereo image in the distance measuring area, which is a predetermined small area in the imaging screen of the image sensor In order to obtain the in-focus position where the high frequency component of the signal is maximized, automatic focus adjustment means for moving and controlling the position of the focus lens, position data acquisition means for acquiring position data of the ranging area in the imaging screen, Subject estimation distance calculation means for calculating an estimated distance from the focus lens focus position to the subject, and a basic scene for generating a pseudo-stereoscopic image signal from a non-stereoscopic image signal Based on a basic depth model generating means for generating a plurality of basic depth model type images and a first control signal indicating a composition ratio of the plurality of basic depth model type images, the basic depth model generating means supplies the images. Based on a depth model combining means for generating a depth model image by combining a plurality of basic depth model type images, and a second control signal indicating a weighting coefficient for weighting the non-stereo image signal, Depth estimation from weighting means for multiplying the image signal by a weighting coefficient indicated by the second control signal, the non-stereo image signal weighted by the weighting means, and the depth model image generated by the depth model synthesis means Depth estimation data generation means for generating data, and a third control signal indicating depth and congestion When shifting the texture of the non-stereo image signal based on the adjusted depth estimation data and generating the pseudo stereo image signal, and calculating and outputting the first to third control signals, respectively. And control signal calculation means for varying at least one of the first and third control signals to be calculated according to the position data of the distance measurement area and the estimated distance to the subject. And

  In order to achieve the above object, the pseudo-stereoscopic image generation apparatus according to the present invention is configured to perform a peristaltic movement of an imaging element that photoelectrically converts an optical image of a subject formed on an imaging surface through an optical system to obtain a non-stereoscopic image signal. Camera shake detection means for detecting camera shake information composed of size information, panning and tilting size information, and a basic scene for generating a pseudo-stereoscopic image signal from a non-stereoscopic image signal obtained by an image sensor And a basic depth model generating means for generating a plurality of basic depth model type images and a first control signal indicating a synthesis ratio of the plurality of basic depth model type images. Depth model compositing means that generates a depth model image by combining multiple basic depth model type images, and weighting of non-stereo image signals A weighting means for multiplying the non-stereo image signal by a weighting coefficient indicated by the second control signal based on the second control signal indicating the weighting coefficient, a non-stereoscopic image signal weighted by the weighting means, and a depth Depth estimation data generating means for generating depth estimation data from the depth model image generated by the model synthesizing means, and depth estimation data in which the depth and congestion are adjusted by a third control signal indicating depth and congestion When shifting the texture of the non-stereo image signal to generate a pseudo stereo image signal and calculating and outputting each of the first to third control signals, the camera shake information is used. Control signal calculating means for varying the value of at least one of the first to third control signals to be calculated. To.

  In order to achieve the above object, the pseudo-stereoscopic image generation apparatus of the present invention provides an optical axis of an imaging element that obtains a non-stereoscopic image signal by photoelectrically converting an optical image of a subject formed on an imaging surface through an optical system. A roll angle detection means for detecting a surrounding roll angle, and a plurality of basic depth model type images indicating a scene serving as a basis for generating a pseudo-stereoscopic image signal from a non-stereoscopic image signal obtained by an image sensor. Based on the first control signal indicating the synthesis ratio of the basic depth model generating means and the images of the plurality of basic depth model types, the images of the plurality of basic depth model types supplied from the basic depth model generating means are synthesized. Based on the depth model synthesis means for generating the image of the depth model and the second control signal indicating the weighting coefficient for weighting the non-stereo image signal, Depth estimation from weighting means for multiplying the image signal by a weighting coefficient indicated by the second control signal, the non-stereo image signal weighted by the weighting means, and the depth model image generated by the depth model synthesis means Based on depth estimation data generating means for generating data and depth estimation data in which the depth and congestion are adjusted by the third control signal indicating the depth and congestion, the texture of the non-stereoscopic image signal is shifted and simulated. When calculating and outputting the texture shift means for generating the stereoscopic image signal and the first to third control signals, the first and the third calculated using the roll angle detection information detected by the roll angle detection means Control signal calculation means for varying the value of at least one of the third control signals.

  Furthermore, in order to achieve the above object, the camera of the present invention is a camera having an image sensor that photoelectrically converts an optical image of a subject imaged on an imaging surface through an optical system to obtain a non-stereoscopic image signal. Between the adjacent two frames of the non-stereoscopic image signal output from the subject estimated distance calculating means for calculating the estimated distance from the subject to the subject and the image sensor that photoelectrically converts the optical image of the subject imaged on the imaging surface through the optical system A scene that is a basis for generating a pseudo-stereoscopic image signal from a non-stereoscopic image signal that is output from an imaging element and a non-stereoscopic image signal that is output from an imaging device is shown based on the signal change and the estimated distance Based on a basic depth model generating means for generating a plurality of basic depth model type images and a first control signal indicating a composition ratio of the plurality of basic depth model type images. And a depth model synthesizing unit for synthesizing a plurality of basic depth model type images supplied from the basic depth model generating unit to generate a depth model image, and a weighting coefficient for weighting the non-stereo image signal. Based on the second control signal, the weighting means for multiplying the non-stereo image signal by a weighting coefficient indicated by the second control signal, the non-stereo image signal weighted by the weighting means, and the depth model synthesis means Based on the depth estimation data generated by the depth estimation data generation means for generating depth estimation data, and the depth estimation data in which the depth and the congestion are adjusted by the third control signal indicating the depth and the congestion, Texture shifting means for generating a pseudo stereoscopic image signal by shifting the texture of the stereoscopic image signal; When calculating and outputting each control signal, the value of at least one of the first to third control signals to be calculated using the shooting scene information acquired by the shooting scene information acquisition means And a control signal calculating means for varying.

  According to the present invention, a pseudo-stereoscopic image signal of an optimal depth model is generated for a non-stereoscopic image signal obtained by photographing by correcting a control signal from photographing parameters obtained for each screen. Accordingly, it is possible to suppress a sense of discomfort and fatigue of a user who views the pseudo stereoscopic image signal, and to generate a pseudo stereoscopic image signal having a more powerful and realistic feeling.

1 is a block diagram of a first embodiment of a video camera including a pseudo stereoscopic image generation device of the present invention. FIG. It is a block diagram of an example of the 2D3D conversion part in FIG. It is a block diagram of an example of the depth estimation data generation part in FIG. It is a block diagram of an example of the stereo pair production | generation part in FIG. It is an example of the three-dimensional structure of the image of basic depth model type A. It is an example of the three-dimensional structure of the image of basic depth model type B. It is an example of the three-dimensional structure of the image of basic depth model type C. It is a figure which shows an example of the determination conditions of the composite ratio in the composite ratio determination part in FIG. It is a block diagram of an example of the depth model synthetic | combination part in FIG. It is a block diagram of 2nd Embodiment of the video camera provided with the pseudo stereo image production | generation apparatus of this invention. It is a block diagram of 3rd Embodiment of the video camera provided with the pseudo stereo image production | generation apparatus of this invention.

  Next, each embodiment of the present invention will be described with reference to the drawings.

<First Embodiment>
FIG. 1 shows a block diagram of a first embodiment of a camera provided with a pseudo stereoscopic image generating apparatus according to the present invention. In the following description of each embodiment, the camera is described as a video camera, but is not limited to a video camera. The video camera 100 according to the present embodiment is a handy video camera having an optical system including a zoom lens 101, a focus lens 102, a diaphragm 103, and an image sensor 104.

  The zoom lens 101 is controlled to move independently from each other in the direction approaching or moving away from the imaging surface of the image sensor 104 by the zoom lens driving unit 105 and the focus lens 102 by the focus lens driving unit 106, respectively. The aperture value of the aperture 103 is controlled by the aperture driver 107. The image sensor 104 is composed of a CCD (Charge Coupled Devise) or CMOS (Complementary Metal Oxide Semiconductor) sensor, and has an imaging surface in which a large number of pixels are arranged in a matrix in the row direction and the column direction. The image sensor 104 photoelectrically converts an optical image of a subject formed on the imaging surface and outputs an imaging signal.

  The zoom lens driving unit 105 has a function of detecting the movement position of the zoom lens 101. Similarly, the focus lens driving unit 106 has a function of detecting the movement position of the focus lens 102. The aperture driving unit 107 has a function of detecting an aperture amount (aperture value).

  A CPU (Central Processing Unit) 108 constitutes a subject face size information acquisition unit, a lens position acquisition unit for acquiring each position of the zoom lens 101 and the focus lens 102, and an optical system in the present invention. Optical system information acquisition means for acquiring the position of the zoom lens 101, the position of the focus lens 102, and the aperture value of the diaphragm 103, subject estimation distance calculation means for calculating the estimated distance from the focus position of the focus lens 102 to the subject, and estimation A depth-of-field calculating means for calculating the depth of field, a position data acquiring means for acquiring position data of a distance measuring area in the imaging screen, and a control signal calculating means for calculating the control signals CTL1 to CTL3 are configured. The CPU 108 also constitutes an automatic focus adjustment unit that controls the movement of the focus lens 102 to the in-focus position together with the focus lens driving unit 106.

  That is, the CPU 108 moves the zoom lens 101 detected by the zoom lens driving unit 105, moves the focus lens 102 detected by the focus lens driving unit 106, and the aperture amount (aperture) detected by the aperture driving unit 107. Value) is input, and the input information is processed according to a program and a table stored in the memory 109. Further, the CPU 108 generates control signals CTL1, CTL2, and CTL3, which will be described later, from imaging information obtained for each scene of the imaging signal.

  The memory 109 is a numerical value table (hereinafter referred to as “the focus lens 102”) for estimating the distance from the focus position of the focus lens 102 to the subject from the combination of the movement position of the zoom lens 101 and the movement position of the focus lens 102 in the focused state. A subject distance estimation table). Further, the memory 109 is a numerical value table (hereinafter referred to as “depth of field table”) for estimating a so-called depth of field from the estimated value of the aperture value, the moving position of the zoom lens 101, and the distance from the subject. ) Is stored.

  In addition, the video camera 100 includes an A / D converter 110 that converts an image pickup signal into a digital signal, a signal processing unit 111 that performs predetermined signal processing on the digital signal, and outputs image data. An image storage unit 112 that stores image data, an image processing unit 113 that performs image processing such as encoding on the image data, and a recording / reproducing interface (I / F) unit 114 are included.

  Furthermore, the video camera 100 includes a pseudo stereoscopic image generation unit (hereinafter referred to as “2D3D conversion unit”) 115 and a display I / F unit 116. As will be described in detail later, the 2D3D conversion unit 115 estimates depth information from a captured image that is a non-stereo image based on the control signals CTL1, CTL2, and CTL3 generated by the CPU 108, and there is almost no sense of incongruity in each scene. A pseudo stereoscopic image is generated. The display I / F unit 116 displays a pseudo stereoscopic image. The 2D3D conversion unit 115 is directly connected to the CPU 108 described above, and is connected to the CPU 108 via the bidirectional image data bus 117 together with the image storage unit 112, the image processing unit 113, and the recording / playback I / F unit 114. Yes.

  In addition, the video camera 100 includes an operation unit 118. The operation unit 118 includes a touch panel. When a user touches a finger on a position on the touch panel, a position corresponding to the position on the shooting screen is selected as a distance measurement area. In the video camera 100, a zoom lens 101, a focus lens 102, an aperture 103, an image sensor 104, a zoom lens driving unit 105, a focus lens driving unit 106, an aperture driving unit 107, a CPU 108, a memory 109, a recording / playback I / F unit 114, The 2D3D conversion unit 115 configures the pseudo-stereoscopic image generation apparatus according to the present embodiment.

  The video camera 100 including the above-described components includes a subject shooting in a desired shooting mode, recording of captured image data onto a recording medium, generation of a pseudo stereoscopic image based on image data reproduced from the recording medium, and a pseudo stereoscopic image. Is appropriately selected and performed.

  Therefore, first, operations at the time of shooting by the video camera 100 and at the time of recording the shot image data on a recording medium will be described. At the time of shooting, incident light from a subject that sequentially passes through the zoom lens 101, the focus lens 102, and the aperture 103 is formed as an optical image on the imaging surface of the image sensor 104, photoelectrically converted, and output as an imaging signal.

  The A / D converter 110 converts the image signal output from the image sensor 104 into image data that is a digital signal, and supplies the image data to the signal processing unit 111. The signal processing unit 111 performs color process processing including pixel interpolation processing on the input image data to generate a digital luminance signal (Y) and chrominance signals (Cb, Cr). Remove noise. The image storage unit 112 sequentially stores the digital luminance signal (Y) and color difference signals (Cb, Cr) generated by the signal processing unit 111. The image storage unit 112 stores signals for several frames.

  The image processing unit 113 accesses the image storage unit 112 via the image data bus 117 and converts the image data stored in the image storage unit 112 into a known MPEG-2 (Moving Picture Experts Group 2) system or MPEG-4. Encoding is performed by AVC (Moving Picture Experts Group 4 Advanced Video Coding) method.

  The encoded image data of the subject obtained by photographing in this way is recorded on a recording medium. That is, the recording / reproducing I / F unit 114 takes in the image data obtained by encoding the captured image of the subject by the image processing unit 113 via the image data bus 117, and records the image data on a recording medium (not shown). To do. However, not only encoded image data as described later but also predetermined data is multiplexed and recorded on the encoded image data on the recording medium. Examples of the recording medium include a magnetic disk, an optical disk, a semiconductor storage medium, and a magnetic tape. The storage medium can be an externally removable storage medium or a built-in storage medium.

  Here, the video camera 100 has an automatic focus adjustment (also referred to as AF) shooting mode. Several methods are known for the AF method. Here, the focus lens position where the high frequency component of the luminance signal (Y) of a predetermined small area in the screen is maximized is set as the in-focus position. A method (so-called mountain climbing method) is used. In this hill-climbing method, the high frequency component (hereinafter referred to as a focus evaluation value) of the luminance signal obtained from the image sensor 104 is stored while moving the position of the focus lens 102 over the entire distance measuring range, and the maximum value of the stored values is stored. The focus lens position corresponding to is used as the in-focus position. Once the in-focus position is obtained, the subsequent photographing corresponds to moving image photographing by driving the focus lens 102 in the vicinity of the in-focus position to obtain the maximum value of the high frequency component of the luminance signal.

  In this AF shooting mode, the CPU 108 reads out data of pixels belonging to a predetermined small area from the luminance signal (Y) for one frame stored in the image storage unit 112 via the image data bus 117. Then, the CPU 108 obtains a focus evaluation value by calculating a high frequency component from the pixel data by digital filter calculation. Further, the CPU 108 instructs the focus lens driving unit 106 to move the focus lens 102.

  The predetermined small area is called a distance measuring area. Several methods are known for determining the distance measurement area. The first distance measurement area determination method is a method of fixing the distance measurement area at the center of the shooting screen. The second distance measurement area determination method is a method in which the user selects the distance measurement area at a desired position on the shooting screen. In this method, when the user touches the touch panel included in the operation unit 118, a small area having a predetermined size centering on a position corresponding to the touched position on the shooting screen is selected as the distance measurement area.

  The third distance measurement area determination method tracks a subject existing in a small area having a predetermined size centering on a position corresponding to the touched position on the shooting screen when the user touches the touch panel. Thus, the tracking area is used as a ranging area. For example, when shooting a dog walking around the meadow, touch the position corresponding to the position of the dog in the shooting screen, and even if the dog walks around after that, the dog's position is automatically tracked and measured Let it be an area.

  The CPU 108 reads out the luminance signal (Y) and the color difference signals (Cb, Cr) for one frame stored in the image storage unit 112, and uses these signals in the shooting screen corresponding to the touched position on the touch panel. The feature amount of the signal in the small area centering on the position of is extracted. As the feature amount, a color distribution, a shape of a high frequency component of a luminance signal, or the like is used. In the method of tracking a moving object in the ranging area, a position having a similar feature amount is searched using the luminance signal (Y) and color difference signals (Cb, Cr) of each subsequent frame while the object is moving. By doing so, the ranging area is determined.

  The fourth distance measurement area determination method is a method of automatically detecting the face of a subject person on the shooting screen and setting a small area having a predetermined size centered on the position of the face as the distance measurement area. It is. Since the human face detection means itself is known, the description thereof is omitted. When a plurality of faces are detected, a small area having a predetermined size centered on the position of the face determined to be the main subject is set as the distance measurement area.

  As the distance measurement area determination method, for example, there is a method in which the face corresponding to the touched position is determined as the face of the main subject when the user touches the touch panel. Alternatively, when the pre-registered person's feature amount is stored in the memory 109 by the personal authentication function, the registered person's face is judged to be the face of the main subject rather than the unregistered person, and the registered plural When a person's face is detected, there is a method of determining the face of the main subject according to the registration order set by the user at the time of registration.

  In addition, when the position of the face is set as the distance measurement area, the size information of the face is also acquired. The face size information is, for example, information in which the shooting screen area (number of pixels) of the face of the subject person detected in the shooting screen is associated with the position (zoom ratio) of the zoom lens 101 at the time of shooting. is there. In the AF shooting mode described above, the focus position of the distance measurement area changes from moment to moment, and in the tracking method and face detection method, the position of the distance measurement area changes from moment to moment.

  At the time of shooting, the zoom lens 101 can be moved in accordance with a user operation on the operation unit 118 to enlarge or reduce the shooting screen. In this case, the CPU 108 instructs the zoom lens driving unit 105 to move the zoom lens 101 in response to the zoom lens operation signal input from the operation unit 118.

  At the time of shooting, the aperture amount (aperture value) can be changed according to the user's operation on the operation unit 118. In this case, the CPU 108 instructs the diaphragm drive unit 107 in response to the diaphragm control signal from the operation unit 118 to change the diaphragm amount of the diaphragm 103. The CPU 108 may automatically control the aperture amount according to the brightness of the subject.

  The moving position of the zoom lens 101, the moving position of the focus lens 102, and the amount of aperture of the diaphragm 103 are respectively acquired by the zoom lens driving unit 105, the focus lens driving unit 106, and the aperture driving unit 107, and the acquired information is sent to the CPU 108. . The CPU 108 estimates the distance from the in-focus position of the focus lens 102 to the subject using the acquired information and the subject distance estimation table stored in the memory 109 (that is, calculates the estimated distance to the subject). ). Further, the CPU 108 estimates the depth of field using the acquired information and the depth of field table stored in the memory 109.

  Subsequently, the CPU 108 determines the position data in the shooting area of the ranging area for each frame, the estimated distance data to the subject, the estimated depth of field data, and the data on whether or not the subject is a face. Includes the face size data and outputs the various data to the recording / playback I / F unit 114. The recording / reproducing I / F unit 114 multiplexes the image data encoded by the image processing unit 113 and the various data generated by the CPU 108 in correspondence with each other, and records them on a recording medium (not shown). The method for multiplexing the above various data into the encoded image data is not particularly limited. For example, the above various data may be embedded in the encoded image data as user data by a known method. May be.

  It should be noted that the data of the position of the ranging area, the estimated distance to the subject, and the estimated depth of field need not be multiplexed every frame, may be every other frame, and there is a change larger than a predetermined threshold. It may be limited to the case.

  As for the estimated distance to the subject and the estimated depth of field, the moving position of the zoom lens 101, the moving position of the focus lens 102, and the aperture amount of the aperture 103 may be encoded instead. If the data corresponding to the subject distance estimation table and the depth of field table are multiplexed and recorded together with them, the estimated distance to the subject and the estimated depth of field can be calculated on the reproduction side. Further, if limited to self-recording / reproduction, the subject distance estimation table and the depth of field table need not be recorded.

  In this embodiment, the estimated distance to the subject is obtained by the hill-climbing method. However, the present invention is not limited to this, and a method of irradiating infrared rays and observing the reflected wave, or a so-called phase difference detection method is used. May be.

  Next, an operation at the time of generating a pseudo stereoscopic image based on image data reproduced from the recording medium of the present embodiment will be described.

  During reproduction, the recording / playback I / F unit 114 reads multiplexed data recorded on a recording medium (not shown), and the encoded image data of the multiplexed data is sent to the image processing unit 113 via the image data bus 117. Supply. In addition, the recording / playback I / F unit 114, among the read multiplexed data, other than the encoded image data, the position of the distance measurement area, the estimated distance to the subject, the estimated depth of field, whether the subject is a face or not And various data such as face size data in the case of a face are supplied to the CPU 108 via the image data bus 117.

  The image processing unit 113 decodes input encoded image data to obtain a decoded luminance signal (Y) and a decoded color difference signal (Cb, Cr) as digital values, and stores them in the image storage unit 112. The image storage unit 112 reads the stored decoded luminance signal (Y) and decoded chrominance signal (Cb, Cr) by rearranging the frames in order of time, and supplies them to the 2D3D conversion unit 115.

  On the other hand, the CPU 108 stores various data such as the position of the distance measurement area, the estimated distance to the subject, the estimated depth of field, whether or not the subject is a face, and the face size data in the case of a face. Based on this, as described later, control signals CTL1, CTL2, and CTL3 are calculated for each frame and supplied to the 2D3D conversion unit 115. At this time, the CPU 108 matches the timing at which a certain frame in the decoded luminance signal (Y) and the decoded color difference signal (Cb, Cr) (hereinafter referred to as a decoded image signal a) is input to the 2D3D conversion unit 115. The control signals CTL1 to CTL3 corresponding to the frame are supplied to the 2D3D conversion unit 115.

  Here, the control signals CTL1 to CTL3 are signals that are automatically generated for each frame by the CPU 108 when the pseudo-stereoscopic image obtained without the control signal becomes unnatural. Note that the control signals CTL1 to CTL3 may be generated for each field.

  The control signal CTL1 is a signal for controlling a synthesis ratio of each image of basic depth model types A, B, and C, which will be described later, and is a synthesis ratio k1 of the basic depth model type A and a synthesis ratio k2 of the basic depth model type B. And a parameter indicating the combination ratio k3 of the basic depth model type C. Since the total value of the above-described three synthesis ratios k1 to k3 is always “1”, the control signal CTL1 transmits, for example, any two synthesis ratios among the three synthesis ratios k1 to k3, and the remaining One composition ratio can be generated by subtracting the sum of any two composition ratios to be transmitted from “1” in the 2D3D conversion unit 115.

  Further, the control signal CTL2 indicates a weighting coefficient for weighting only the red signal (R signal) component in the decoded image signal a which is a non-stereo image signal obtained by decoding by the image processing unit 113. It is a control signal. This control signal CTL2 can suppress an unnatural pseudo stereoscopic image even when the luminance difference of the decoded image signal a is strong.

  Further, the control signal CTL3 is a control signal including a parameter indicating depth and a parameter indicating congestion. Convergence means that the line of sight of both eyes of the user of the pseudo-stereoscopic image is substantially parallel to a distant view, and that the line of sight of both eyes can be inverted for a close view.

  In FIG. 1, the 2D3D conversion unit 115 receives the decoded video signal a supplied from the image storage unit 112 and the control signals CTL1 to CTL3 supplied from the CPU 108 as input signals. Alternatively, the depth estimation data d is generated based on the control signals CTL1 and CTL2 and output to the CPU 108, and the pseudo stereoscopic image signal is generated based on the decoded video signal a and the control signals CTL1 to CTL3.

  Next, the configuration and operation of the 2D3D conversion unit 115 will be described in detail.

  2 is a block diagram of an example of the 2D3D conversion unit 115, FIG. 3 is a block diagram of an example of the depth estimation data generation unit in FIG. 2, and FIG. 4 is a block of an example of the stereo pair generation unit 300 in FIG. The figure is shown.

  As illustrated in FIG. 2, the 2D3D conversion unit 115 includes a depth estimation data generation unit 200 and a stereo pair generation unit 300. The 2D3D conversion unit 115 outputs the decoded image signal a, which is a non-stereo image signal supplied from the image storage unit 112, as it is, for example, as the right-eye video signal e2 without being processed, and also includes the depth estimation data generation unit 200 and the stereo It supplies to the pair production | generation part 300, respectively.

  The depth estimation data generation unit 200 generates depth estimation data d from the control signals CTL1 and CTL2 supplied from the CPU 108 and the decoded image signal a supplied from the image storage unit 112, and sends the depth estimation data d to the CPU 108 and the stereo pair generation unit 300. Supply. The stereo pair generation unit 300 generates a left-eye video signal e1 of a pseudo stereoscopic image signal from the depth estimation data d, the control signal CTL3 supplied from the CPU 108, and the decoded image signal a supplied from the image storage unit 112. Output.

  As shown in FIG. 3, the depth estimation data generation unit 200 includes an image input unit 201, a high-frequency component evaluation unit 202 at the top of the screen, a high-frequency component evaluation unit 203 at the bottom of the screen, a composition ratio determination unit 204, A switch 205, a depth model synthesis unit 206 as a depth model synthesis unit, frame memories 207, 208 and 209 as basic depth model generation units, a control signal determination unit 210, a weighting unit 211 as a weighting unit, It is comprised from the addition part 212 which is the depth estimation data production | generation means which produces | generates depth estimation data.

  The image input unit 201 includes a frame memory. After temporarily storing the decoded image signal a for one frame, which is a non-stereo image signal, the decoded image signal a for one frame is evaluated as a high-frequency component at the top of the screen. The unit 202 and the high-frequency component evaluation unit 203 at the bottom of the screen are respectively supplied, and only the red signal (R signal) component in the decoded image signal a is supplied to one input terminal of the weighting unit 211 serving as a weighting unit. The other input terminal of the weighting unit 211 is supplied with a control signal CTL2 from a control signal determination unit 210 that is a control signal determination unit.

  The high-frequency component evaluation unit 202 at the top of the screen calculates a high-frequency component in an area corresponding to about 20% of the top of the screen in the decoded image signal a for one frame, and calculates it as a high-frequency component evaluation value at the top of the screen. Then, the high frequency component evaluation unit 202 at the top of the screen supplies the high frequency component evaluation value at the top of the screen to the synthesis ratio determination unit 204. A high-frequency component evaluation unit 203 at the bottom of the screen calculates a high-frequency component in an area corresponding to approximately 20% of the bottom of the screen in the decoded image signal a for one frame, and calculates it as a high-frequency component evaluation value at the bottom of the screen. To do. Then, the high frequency component evaluation unit 203 at the bottom of the screen supplies the high frequency component evaluation value at the bottom of the screen to the synthesis ratio determination unit 204.

  On the other hand, the frame memory 207 stores the basic depth model type A, the frame memory 208 stores the basic depth model type B, and the frame memory 209 stores the basic depth model type C. The images of these basic depth model types A to C are images of scenes serving as a basis for generating pseudo stereoscopic image signals by generating depth estimation data based on non-stereo image signals, respectively.

  That is, the basic depth model type A image is a depth model image having a spherical concave surface, and an image having a three-dimensional structure as shown in FIG. In many cases, this basic depth model type A image is used. This is because, in a scene in which no object exists, setting the center of the screen to the farthest distance can provide a three-dimensional effect with less discomfort and a comfortable depth feeling.

  In addition, the basic depth model type B image is obtained by replacing the upper part of the basic depth model type A image with an arch-shaped cylindrical surface instead of a spherical surface. It is an image of a model in which a cylindrical surface (axis is a vertical direction) and a lower surface is a concave surface (spherical surface).

  Furthermore, the basic depth model type C image is a cylindrical surface that has a three-dimensional structure as shown in FIG. 7, with the upper part being a plane, the lower part being continuous from the plane, and going downwards toward the front. The upper part is a plane image and the lower part is a cylindrical surface (the axis is the horizontal direction). The images of the basic depth model types A to C stored in the frame memories 207 to 209 constituting the basic depth model type generation unit are supplied to the depth model synthesis unit 206.

  The composition ratio determination unit 204 includes a high frequency component evaluation value at the top of the screen supplied from the high frequency component evaluation unit 202 at the top of the screen and a high frequency component evaluation at the bottom of the screen supplied from the high frequency component evaluation unit 203 at the bottom of the screen. Based on the value, the composition ratio k1 of the basic depth model type A, the composition ratio k2 of the basic depth model type B, and the composition ratio of the basic depth model type C by a predetermined method without considering the scene of the image k3 is automatically calculated and supplied to the switch 205, which is a control signal determination means, as a composite ratio signal COM. The total value of the three synthesis ratios k1 to k3 is always “1”.

  FIG. 8 shows an example of conditions for determining the composition ratio. FIG. 8 shows a high-frequency component evaluation value at the top of the screen indicated by the horizontal axis (hereinafter abbreviated as the high-frequency component evaluation value at the top) and a high-frequency component evaluation value at the bottom of the screen indicated by the vertical axis (hereinafter, the high frequency component at the bottom It is shown that the composition ratio is determined based on the balance between each value of the component evaluation value (abbreviated as component evaluation value) and the predesignated values tps, tpl, bms, and bml. The determination condition of the synthesis ratio is a known determination condition disclosed by the present applicant in Japanese Patent No. 4214976, but is not limited thereto.

  In FIG. 8, regions where a plurality of types are described are synthesized linearly according to the high frequency component evaluation value. For example, in FIG. 8, the area of “type A / B” is the value of the basic depth model type A at the ratio of (upper high-frequency component evaluation value) and (lower high-frequency component evaluation value) as follows. The ratio between type A and type B, which is the value of basic depth model type B, is determined, and type C, which is the value of basic depth model type C, is not used for determining the ratio.

typeA: typeB: typeC
= (Upper high-frequency component evaluation value-tps): (tp1-Lower high-frequency component evaluation value): 0
In FIG. 8, in the region of “type A / B / C”, the average of type A / B and type A / C is adopted, and the value of type A / B / C is determined as follows.

typeA: typeB: typeC
= (Upper high-frequency component evaluation value-tps) + (Lower high-frequency component evaluation value-bms): (tpl-
Upper high band component evaluation value): (bml-lower high band component evaluation value)
The synthesis ratios k1, k2, and k3 are calculated by the following equations.

k1 = type1 / (typeA + typeB + typeC)
k2 = type2 / (typeA + typeB + typeC)
k3 = type3 / (typeA + typeB + typeC)
Returning to FIG. The switch 205 selects the control signal CTL1 when the control signal CTL1 is supplied from the CPU 108 and supplies the selected control signal CTL1 to the depth model synthesis unit 206. When the control signal CTL1 is not supplied, the switch 205 is supplied from the synthesis ratio determination unit 204. The combination ratio signal COM is selected. The switch 205 supplies the selected signal to the depth model synthesis unit 206 which is a synthesis unit.

  The depth model synthesis unit 206 synthesizes the images of the basic depth model types A to C at the ratio indicated by the synthesis ratios k1 to k3 indicated by the control signal CTL1 or the synthesis ratio signal COM supplied from the switch 205 and the synthesized depth model. An image signal is generated.

  FIG. 9 is a block diagram illustrating an example of the depth model synthesis unit 206. As shown in the figure, the depth model combining unit 206 includes the combination ratios k1, k2, and k3 indicated by the control signal CTL1 or the combination ratio signal COM selected by the switch 205, and the basic depth from the frame memories 207, 208, and 209. The image signals of model types A, B, and C are separately multiplied by multipliers 2061, 2062, and 2063, the three multiplication results are added by an adder 2064, and the obtained addition result is obtained as depth estimation data. It is the structure which outputs to the addition part 212 which is a production | generation means.

  In FIG. 3, the control signal determination unit 210 determines whether or not the control signal CTL2 is supplied from the CPU. When the control signal CTL2 is supplied, the control signal determination unit 210 supplies the control signal CTL2 to the weighting unit 211. On the other hand, when the control signal CTL2 is not supplied, the control signal CTL2 including a parameter corresponding to a weighting coefficient set in the control signal determination unit 210 in advance is supplied to the weighting unit 211.

  The weighting unit 211 calculates a weighting coefficient for setting the maximum value included in the control signal CTL2 supplied from the control signal determination unit 210 to “1” and the R signal component of the decoded image signal a supplied from the image input unit 201. Multiplication is performed to weight the R signal component of the decoded image signal a. The R signal component is calculated in the image input unit 201 from the luminance signal (Y) and the color difference signal Cr by an operation such as R = Y + Cr.

  One of the reasons for using the R signal component is that the size of the R signal component is the same as that of the original image in an environment close to direct light and under a condition in which the change in the brightness level (brightness) of the texture is not large. This is based on an empirical rule that there is a high probability of matching with the irregularities of A texture is an element constituting an image and is composed of a single pixel or a group of pixels.

  Further, another reason for using the R signal component is that red and warm colors are advanced colors in chromaticity, and the depth is recognized in front of the cold color system. It is possible to emphasize the stereoscopic effect.

  Note that red and warm colors are forward colors, whereas blue is a backward color, and the depth is recognized deeper than the warm color system. Therefore, it is possible to enhance the stereoscopic effect by arranging the blue portion in the back. Furthermore, it is also possible to enhance the stereoscopic effect by using both in combination and arranging the red part in front and the blue part in the back.

  The adding unit 212 generates depth estimation data d from the image signal of the combined depth model supplied from the depth model combining unit 206 and the R signal component of the weighted decoded image signal a supplied from the weighting unit 211. . For example, the depth estimation data d is generated by superimposing the R signal component of the weighted decoded image signal a supplied from the weighting unit 211 on the image signal of the combined depth model supplied from the depth model combining unit 206. When the superposed value exceeds a predetermined number of bits assigned to the depth estimation data d, the number is limited to a predetermined number of bits. The generated depth estimation data d is supplied to the stereo pair generation unit 300. It is also supplied to the CPU 108.

  Next, the configuration and operation of the stereo pair generation unit 300 will be described. FIG. 4 shows a block diagram of an example of the stereo pair generation unit 300. A control signal determination unit 301 serving as a control signal determination unit determines whether or not the control signal CTL3 is supplied from the CPU. When the control signal CTL3 is not supplied, two parameters representing the congestion and the depth set in the control signal determination unit 301 in advance are supplied to the texture shift unit 302. When the control signal CTL3 is supplied, two parameters representing the congestion and the depth included in the control signal CTL3 are supplied to the texture shift unit 302.

  The texture shift unit 302 serving as a texture shift unit is based on the supplied decoded image signal a, the depth estimation data d, and the two parameters representing the congestion and depth from the control signal determination unit 301. An image signal of another viewpoint is generated. For example, the texture shift unit 302 generates an image signal whose viewpoint is moved to the left with reference to the viewpoint when the decoded image signal a is displayed on the screen. In that case, when the texture shift unit 302 displays the texture as a foreground for the user, the closer the image is, the closer it is to the inner side (nose side) of the user, so that the image is moved to the right side of the screen by an amount corresponding to the depth. Generate a signal. Further, when displaying the texture as a distant view for the user, the texture shift unit 302 generates an image signal in which the texture is moved to the left side of the screen by an amount corresponding to the depth because the farther the image is visible to the outside of the user.

  Here, the depth estimation data d for each pixel is represented by an 8-bit value Dd. The texture shift unit 302 sequentially increases the texture of the decoded image signal a corresponding to the Dd by (Dd−m) / n pixels in order from the smallest value of Dd (that is, the one located at the back of the screen). An image signal shifted to is generated. The above m is a parameter (congestion value) representing a feeling of popping out, and the above n is a parameter (depth value) representing a depth.

  For the user, a texture having a small value Dd indicating the depth estimation data d appears on the back side of the screen, and a texture having a large value Dd indicating the depth estimation data d appears on the front side of the screen. The value Dd indicating the depth estimation data d, the congestion value m, and the depth value n are values in the range of 0 to 255. For example, the value preset in the control signal determination unit 301 is the congestion value m = 200, the depth The value n = 20.

  The occlusion compensation unit 303 performs occlusion compensation on the image signal of another viewpoint output from the texture shift unit 302 and supplies the occlusion compensated image signal to the post processing unit 304. Occlusion refers to a portion where no texture exists due to a change in the positional relationship in the image as a result of shifting the texture. The occlusion compensation unit 303 fills the occlusion location with the original decoded image signal a corresponding to the texture-shifted image signal. In addition, well-known literature (Kunio Yamada, Kenji Mochizuki, Kiyoharu Aizawa, Takahiro Saito: “Occlusion compensation based on texture statistics of images divided by the region competition method”, Journal of the Institute of Image Information Science, Vol.56, No.5 , pp. 863 to 866 (2002.5)), the occlusion may be compensated.

  The post-processing unit 304, which is a post-processing unit, performs post-processing such as smoothing and noise removal on the image signal that has been occlusion-compensated by the occlusion compensation unit 303, if necessary, using a known method. The signal e1 is output. The 2D3D conversion unit 115 sets the decoded image signal a reproduced from the recording medium by the recording / playback I / F unit 114 as the right-eye image signal e2.

  In FIG. 1, a display I / F unit 116 receives a right-eye image signal e2 output from the 2D3D conversion unit 115 and a post-processed left-eye image signal e1 as input signals, and displays the input signals for stereo image display. Output to a corresponding monitor (not shown). Thereby, the user can see a stereo image. As described above, the generation of the pseudo stereoscopic image and the display of the pseudo stereoscopic image based on the image data reproduced from the recording medium are performed.

  Next, the operation of the main part of this embodiment will be described in more detail.

  When the user converts a non-stereoscopic image captured by the video camera 100 into a pseudo-stereoscopic image, the control signals CTL1 to CTL3 for automatically adjusting the pseudo-stereoscopic image generation parameters without the user becoming a producer. The generation method will be specifically described.

  Of the multiplexed data read out from the recording medium (not shown) by the recording / playback I / F unit 114, the CPU 108 stores the position of the ranging area, the estimated distance to the subject, the estimated depth of field, and whether the subject is a face. In the case of a face, various data such as face size data are acquired. Then, the CPU 108 calculates the control signals CTL1 to CTL3 from the acquired data and the depth estimation data d supplied from the 2D3D conversion unit 115 as follows.

  The CPU 108 determines the basic depth model type when the position of the distance measurement area on the shooting screen is the center in the horizontal direction of the screen and the estimated distance to the subject is smaller than the predetermined threshold 1 than when it is not. The control signal CTL1 is generated so that the mixing ratio of C is increased. The reason for this is that a subject at a short distance is the main subject, and is often a portion to be watched during playback viewing. Therefore, in a model arranged farther toward the center of the screen, such as the basic depth model types A and B, the gaze portion is far away, and there is a sense of incongruity.

  Further, when the subject is a face and the normalized face size is larger than a predetermined threshold 2, the CPU 108 determines the probability of a bust shot that is a shot that captures the entire face including the top from the chest of the subject. Therefore, the control signal CTL1 is generated so that the mixing ratio of the basic depth model type C is higher than that in the case where it is not. This is based on the same reason as described above. Similarly, the control signal CTL1 is generated so that the mixing ratio of the basic depth model type C is higher as the estimated depth of field is shallower. This is because there is a sense of incongruity when there is a change in the depth of the out-of-focus portion.

  Further, the CPU 108 determines the basic depth model type B when the position of the ranging area on the shooting screen is the right or left end in the horizontal direction of the screen and the estimated distance to the subject is smaller than the predetermined threshold value 3. The control signal CTL1 is generated so that the mixing ratio is higher than in the case where the mixing ratio is not. The reason for this is that the part to be watched is considered to be the right or left edge in the horizontal direction of the screen, so it is less uncomfortable to use the model with the part in front of the screen and the other part in the back of the screen. is there.

  In addition, when the estimated distance to the subject is larger than the predetermined threshold 4, the CPU 108 generates the control signal CTL1 so that the mixing ratio of the basic depth model type A is higher than that when it is not. The reason is that the pseudo-stereoscopic image is powerful by giving the screen a feeling of spreading.

  In other cases, the control signal CTL1 is not output. Therefore, in cases other than the above, the switch 205 outputs the composite ratio signal COM. Further, when the main subject is a face and the normalized face size is larger than the predetermined threshold value 5, the CPU 108 sets the weighting coefficient included in the control signal CTL2 to a larger value than in other cases. And The reason is that since the skin color portion of the face contains an R signal component, increasing the weighting coefficient places the face closer to the screen than the surroundings, so the impression as a pseudo-stereoscopic image becomes stronger. is there.

  Further, the CPU 108 sets the weighting coefficient included in the control signal CTL2 to a larger value than the other cases as the estimated depth of field becomes deeper. This is because when the estimated depth of field is a deep value, many parts of the screen are often seen without blurring, and the sense of depth is lost when viewed in a two-dimensional image (non-stereoscopic image). This is to give a sense of depth.

  Further, the CPU 108 obtains an average value of the values Dd indicating the depth estimation data d of a plurality of pixels corresponding to the positions in the shooting screen of the distance measurement area among the respective pixels of the depth estimation data d. Let this average value be m. That is, the congestion value m of the control signal CTL3 for controlling the congestion and depth is calculated. Therefore, from the above-described equation for the shift amount (Dd−m) / n pixels, the region corresponding to the position of the distance measurement area on the shooting screen is the region with the shift amount 0. The region where the shift amount is zero in the image data for the right eye and the left eye is made to appear on the actual display surface of the monitor in the stereoscopic display. This is because the main subject exists in the distance measurement area and is considered to be a gaze part when viewing and replaying. Therefore, by always positioning this part on the actual display surface of the monitor, a stable impression can be obtained. The purpose is to give.

  Further, the CPU 108 decreases the depth value n of the control signal CTL3 as the estimated depth of field is deeper. That is, the depth as a stereoscopic image is increased. Conversely, as the estimated depth of field is shallower, the CPU 108 decreases the depth value n of the control signal CTL3. That is, the depth as a stereoscopic image is reduced.

  This is due to the following reason. When the estimated depth of field is a deep value, many image portions in the screen are often shown without being blurred. In this case, when viewed as a two-dimensional image (non-stereoscopic image), a feeling of depth is lost. By supplementing this with 2D3D conversion, a sense of depth can be obtained. Conversely, when the estimated depth of field is a shallow value, the portion of the image that appears to be in focus is often limited, and even when viewed in a two-dimensional image (non-stereoscopic image), the sense of depth Is felt. This is because, if a sense of depth is further added by 2D3D conversion in this state, the user may feel as if the user is forced to the focus position of the eyes, and may feel uncomfortable.

  Further, when the main subject is a face and the normalized face size is larger than the predetermined threshold value 5, the CPU 108 sets the depth value n of the control signal CTL3 as compared to the other cases. Make it smaller. That is, the depth as a stereoscopic image is reduced.

  Note that the values of the control signals CTL1 to CTL3 are generated in consideration of temporal continuity in successive frames. That is, the value change is made gradual in time. This is because a sudden change cannot follow the visual sense of the user viewing the pseudo-stereoscopic image.

  As described above, according to the present embodiment, when a depth information is estimated from a captured image that is a non-stereo image and a pseudo stereoscopic image is generated, a plurality of basic depth model types are generated by the first to third control signals. Values of the first to third control signals from the shooting parameters obtained for each screen when controlling the image synthesis ratio, the weighting coefficient of the non-stereo image signal, the depth value and the convergence value at the time of texture shift. Is automatically changed so that the optimal depth model is obtained, so that the user does not have to set it individually according to the shooting scene etc. It is possible to generate a pseudo stereoscopic image that is closer to an actual image.

  In addition, according to the present embodiment, when the subject is a face, it is possible to give a sense of depth suitable for portrait photography. Furthermore, a stable pseudo-stereoscopic image can be displayed by making the ranging area, which is the focused area, appear to be located on the actual display surface of the monitor. Further, in the present embodiment, since the change of the values of the control signals CTL1 to CTL3 is made gradual in time, the relationship between the blur amount and the depth can be made natural without any sense of incongruity for the user.

<Second Embodiment>
Next, a second embodiment of the present invention will be described. FIG. 10 shows a block diagram of a second embodiment of a video camera provided with a pseudo-stereoscopic image generation apparatus according to the present invention. In the figure, the same components as those in FIG. The video camera 150 of the second embodiment shown in FIG. 10 does not perform 2D3D conversion on the reproduced image data, but performs 2D3D conversion on the captured image signal and records the result. Different from the camera 100.

  The video camera 150 shoots a subject in a desired shooting mode, generates a pseudo stereoscopic image based on the captured image data, records the pseudo stereoscopic image on a recording medium, and displays a pseudo stereoscopic image reproduced from the recording medium. Select as appropriate.

  At the time of shooting, the image storage unit 151 in the video camera 150 stores the digital luminance signal (Y) and color difference signals (Cb, Cr) of the subject image output from the signal processing unit 111 in sequence. Similar to the unit 112, but differs from the image storage unit 112 in that the stored luminance signal (Y) and color difference signals (Cb, Cr) are output to the CPU 152 and the 2D3D conversion unit 115, respectively.

  The CPU 152 reads the luminance signal (Y) and color difference signals (Cb, Cr) for one frame read from the image storage unit 151, the moving position of the zoom lens 101, the moving position of the focus lens 102, and the aperture amount of the diaphragm 103. Using the acquired information such as the subject distance estimation table and the depth-of-field table stored in the memory 109, the position data in the shooting screen of the distance measurement area for each frame, the focus lens, and the like in the CPU 108 Data of the estimated distance from the in-focus position 102 to the subject, estimated data of depth of field, data on whether or not the subject is a face, and data on the size of the face in the case of a face are generated.

  Subsequently, the CPU 152 calculates the control signals CTL1 to CTL3 from the generated various data and the depth estimation data d supplied from the 2D3D conversion unit 115 by the same method as described in the first embodiment. This is supplied to the 2D3D conversion unit 115.

  The 2D3D conversion unit 115 is based on the luminance signal (Y) and color difference signals (Cb, Cr), which are non-stereo image signals, read from the image storage unit 151, and the control signals CTL1 and CTL2 calculated by the CPU 152. Then, the depth estimation data d is generated and output to the CPU 152 by the same method as described in the first embodiment, and based on the non-stereo image signal (Y, Cb, Cr) and the control signals CTL1 to CTL3. The left-eye image signal e1 and the right-eye image signal e2 are generated. Then, the 2D3D conversion unit 115 supplies the generated left-eye image signal e1 and right-eye image signal e2 to the image processing unit 153.

  The image processing unit 153 encodes the left-eye image signal e1 and the right-eye image signal e2 supplied from the 2D3D conversion unit 115 independently by a known MPEG-2 system or MPEG-4 AVC system. Alternatively, the image processing unit 153 uses the left-eye image signal e1 and the right-eye image signal e2 supplied from the 2D3D conversion unit 115 as a set of pseudo-stereoscopic image signals, for example, MPEG-4 MVC (Moving Picture Experts Group 4 Multiview Video Coding). ) Encoding based on the method.

  The recording / playback I / F unit 114 receives the image signal encoded by the image processing unit 153 via the image data bus 117 and records the input encoded image signal on a recording medium (not shown).

  According to the present embodiment, since the pseudo stereoscopic image signal of the photographed subject image can be recorded on the recording medium, the video camera 150 is not limited to the case of reproducing the pseudo stereoscopic image signal from the recording medium. Thus, the pseudo stereoscopic image signal can be reproduced and displayed from the recording medium recorded by the video camera 150.

<Third Embodiment>
Next, a third embodiment of the present invention will be described. FIG. 11 shows a block diagram of a third embodiment of a video camera provided with a pseudo-stereoscopic image generation apparatus according to the present invention. In the figure, the same components as those in FIG. A video camera 170 according to the third embodiment illustrated in FIG. 11 has a configuration in which a camera shake sensor 171 and a posture sensor 172 are added to the configuration of the video camera 100.

  The camera shake sensor 171 is composed of a gyro sensor or the like, measures camera movement, and outputs the measurement result to the CPU 173. The camera shake sensor 171 is also a means for measuring camera panning and tilting operations. When the camera motion is measured with a gyro sensor or the like, if the frequency of the angular velocity component is 3 to 10 Hz, the motion is considered to be caused by camera shake. On the other hand, in the case of panning and tilting operations in which the camera is moved in a certain direction for a certain time, the angular velocity of the direct current component is almost obtained. That is, by using the measurement result of the camera movement, which is an output signal from the camera shake sensor 171, by the CPU 173, it is estimated whether the movement is caused by camera shake, panning, or tilting operation. In any case, the magnitude of shaking is calculated.

  The camera movement may be calculated by a method that does not depend on a gyro sensor or the like. For example, as is well known, the average value of the motion vectors of all the subjects in the screen may be calculated from a certain frame of the shooting screen and the next frame, and the variation of the motion vector for each screen may be measured as the camera motion. .

  The posture sensor 172 is composed of a gravity sensor or the like, and here, measures the roll angle around the optical axis of the image sensor 104 and outputs the measurement result to the CPU 173. The roll angle around the optical axis of the image sensor 104 is 0 ° when the horizontal line is horizontal when the horizontal line is imaged, and + a ° (a is positive when the horizontal line is imaged downward to the right). Number), the case where the horizontal line is photographed to the left is −a ° (a is a positive number).

  The CPU 173 includes a face size information acquisition unit of the present invention, a subject estimation distance calculation unit that calculates an estimated distance from the video camera 170 to the subject, a depth of field calculation unit that calculates an estimated depth of field, and imaging. Position data acquisition means for acquiring position data of a ranging area in the screen, control signal calculation means for calculating control signals CTL1 to CTL3, camera shake detection means, roll angle detection means around the optical axis of the image sensor, and shooting scene information It constitutes acquisition means. Note that the camera shake information detected by the camera shake detection means includes not only the above-described camera shake magnitude information but also panning and tilting magnitude information.

  That is, the CPU 173 determines the position of the distance measurement area, the estimated distance to the subject, the estimated depth of field, the data on whether the subject is a face, the size of the face in the case of a face, as described in the first embodiment. In addition to various data such as data, the camera swing sensor 171 magnitude information of panning and tilting, and the roll angle around the optical axis of the image sensor 104 obtained by the attitude sensor 172 Multiplex information.

  The estimated distance to the subject may be obtained by a hill-climbing method as in the first embodiment, or a method of irradiating infrared rays and observing the reflected wave, or a so-called phase difference detection method may be used. . Further, the estimated distance to the subject may be a distance from the focus lens to the subject, or may be a distance from the other part of the camera to the subject.

  Further, the CPU 173 reads the luminance signal (Y) and the color difference signals (Cb, Cr) for one frame stored in the image storage unit 112, and converts them into hue (H), saturation (S), and brightness (V). Convert. Furthermore, the CPU 173 averages the hue (H), saturation (S), and lightness (V) of the signal in each block in each block obtained by dividing the screen for one frame into, for example, 30 horizontal and 20 vertical. And the calculation result is stored in the memory 174. The memory 174 stores the calculation result of the current one frame and the calculation result of the previous frame. In addition to the operation program for the CPU 173, the memory 174 stores a subject distance estimation table and a depth-of-field table as in the memory 109.

  The CPU 173 determines a shooting scene from these calculation results and the estimated distance to the subject. For example, when the lightness (V) of many blocks at the top of the screen is bright, the hue (H) is blue, and the estimated distance is larger than a predetermined threshold, it is determined as a daytime landscape scene. Also, the CPU 173 compares the brightness (V) of the previous frame with the brightness (V) of the previous frame in the entire screen, and compares the blocks corresponding to the same position in the screen with each other. If the number of blocks whose length is greater than the first threshold is greater than the second threshold, it is determined that there is a lot of movement of the subject and a sports shooting scene is determined. Further, the CPU 173 determines that if the hue (H) of the block at the center of the screen is red or blue, the saturation (S) is larger than the third threshold, and the estimated distance to the subject is smaller than the fourth threshold, The macro shooting scene is determined.

  The CPU 173 also multiplexes the shooting scene information with the above-described vibration magnitude information and roll angle information around the optical axis and various data described in the first embodiment, and the recording / playback I / F. Supplied to the unit 114. The recording / reproducing I / F unit 114 records the supplied multiplexed data and the encoded image data of the corresponding frame in association with a recording medium (not shown).

  Next, a pseudo stereoscopic image signal generation operation based on the reproduction signal of the video camera 170 will be described. The recording / playback I / F unit 114 supplies the encoded image data to the image processing unit 113 via the image data bus 117 among the data read from the recording medium (not shown), while the ranging area other than the encoded image data. Position, estimated distance to the subject, estimated depth of field, data on whether the subject is a face, face size data for a face, camera shake information due to camera shake, panning, tilting The size information, roll angle information about the optical axis of the image sensor, and shooting scene information are supplied to the CPU 173 via the image data bus 117.

  The CPU 173 calculates control signals CTL1 to CTL3 as follows from the data supplied from the recording / playback I / F unit 114 and the depth estimation data d supplied from the 2D3D conversion unit 115. Note that the calculation operation of the control signals CTL1 to CTL3 by the CPU 173 includes the same calculation operation as the calculation operation of the control signals CTL1 to CTL3 by the CPU 108, but since the description thereof has already been described, it is omitted and the control signal CTL1 unique to the present embodiment. -Calculation operation of CTL3 will be described below.

  When the shooting scene information indicates a sports shooting scene, the CPU 173 generates the control signal CTL1 so that the mixing ratio of the basic depth model type C is higher than that when it is not. The reason for this is that in the case of sports shooting scenes, the movement of the shooting target is often intense, and the shooting target moves not only in the center of the screen but also toward the edge of the screen. This is to prevent a sense of discomfort for the user who views the pseudo-stereoscopic image from being greatly changed in the model arranged at the back of the screen toward the center of the screen.

  Further, the CPU 173 generates the control signal CTL1 so that the mixing ratio of the basic depth model type C is higher when the value of the size information of panning and tilting is larger than a predetermined threshold. To do. The reason for this is that when panning, the subject that has been at the left and right edges of the screen until then moves to the center of the screen. In this case, it is unnatural that the sense of depth of the subject is different between when it is at the left and right edges of the screen and when it is in the center of the screen, and this is to prevent such a situation.

  In addition, when the shooting scene information indicates a daytime landscape shooting scene, the CPU 173 generates the control signal CTL1 so that the mixing ratio of the basic depth model type A is higher than that when it is not. The reason is that the pseudo-stereoscopic image is powerful by giving a feeling of expansion of the screen.

  Further, when the shooting scene information indicates a flower macro shooting scene, the CPU 173 sets the weighting coefficient included in the control signal CTL2 to a larger value than in other cases. This is because by increasing the weighting coefficient, a flower image including the R signal component is arranged in front of the screen, and the impression as a pseudo stereoscopic image is strengthened.

  On the other hand, the CPU 173 decreases the weighting coefficient included in the control signal CTL2 as the roll angle information around the optical axis is further away from 0 °. The reason is that as the roll angle information around the optical axis is farther from 0 °, the image is taken with the shooting screen tilted with respect to the horizontal line. This is to prevent this because the posture during viewing is not inclined and there is a sense of incongruity. Similarly, the greater the magnitude of vibration due to camera shake, the smaller the weighting coefficient included in the control signal CTL2. This is because if the stereoscopic effect is emphasized while the screen is shaking, the user's posture during viewing of the image is not inclined, which makes the user feel uncomfortable.

  Further, the CPU 173 increases the value of the depth value n in the control signal CTL3 as the magnitude of vibration due to camera shake increases. That is, the depth as a stereoscopic image is reduced. When vibration due to camera shake is large, the user who sees the image is perceived by the user who sees the image during playback and viewing. Therefore, even when viewing the two-dimensional image (non-stereoscopic image), the user feels drunk. Looking at the pseudo-stereoscopic image obtained by 2D3D conversion in this state, the feeling of getting drunk becomes stronger. In order to prevent this, as the magnitude of vibration due to camera shake increases, the sense of depth as a stereoscopic image is reduced.

  Further, the CPU 173 increases the depth value n of the control signal CTL3 as the roll angle information around the optical axis is further away from 0 °. That is, the depth as a stereoscopic image is reduced. The reason is that, as the roll angle information about the optical axis is further away from 0 °, the image is taken with the shooting screen tilted with respect to the horizontal line, and the displayed image is tilted with respect to the horizontal line of the screen. On the other hand, the height of both eyes of the user viewing the display image often does not follow the tilt of the display image, so the pseudo-stereoscopic image obtained by 2D3D conversion in this state has a great sense of depth. This is to prevent the user from feeling uncomfortable when displayed.

  In addition, when the shooting scene information indicates a sports shooting scene, the CPU 173 increases the depth value n of the control signal CTL3. That is, the depth as a stereoscopic image is reduced. In this case, since the movement of the shooting target is intense, the temporal variation of the parallax between the right-eye image and the left-eye image is large, and when the user views a pseudo-stereoscopic image with a great sense of depth, the user feels tired. This is to prevent it because there is a fear.

  Further, the CPU 173 indicates that the shooting scene information indicates a daytime landscape shooting scene, the magnitude of vibration due to camera shake is smaller than the first predetermined value, the roll angle around the optical axis is close to 0 °, and When the condition that the estimated distance is larger than the second predetermined value is satisfied, the depth value n of the control signal CTL3 is decreased. That is, the depth as a stereoscopic image is increased. The reason is that the presence of the pseudo-stereoscopic image to be generated can be enhanced with respect to the image in this case, thereby enhancing the sense of reality.

  As described above, according to the present embodiment, the control signals CTL1 to CTL3 supplied to the 2D3D conversion unit 115 are automatically controlled using the parameters at the time of shooting. Even if it is not set to, the optimum depth model is automatically generated. In particular, in the present embodiment, as shooting parameters for automatically controlling the control signals CTL1 to CTL3, information on the size of the camera shake due to camera shake, information on the size of panning and tilting, information on the roll angle around the optical axis, and Because it includes shooting scene information, it can be used for shooting scenes such as sports shooting scenes, daytime landscape shooting scenes, and macro shooting scenes of flowers, as well as non-stereoscopic images that are tilted with respect to the horizontal line of the screen, Generates a pseudo-stereoscopic image of the optimal depth model for non-stereoscopic images with large fluctuations, suppresses discomfort and fatigue of the user viewing the pseudo-stereoscopic image, and more powerful and realistic pseudo-stereoscopic images Can be generated and displayed.

  Note that the present invention is not limited to the above-described embodiments. For example, a pseudo-stereoscopic image generation program that causes the CPUs 108, 152, or 173 to execute the pseudo-stereoscopic image generation operations of the above-described embodiments is also included in the present invention. It is. In this case, the pseudo stereoscopic image generation program may be taken into the CPU from a recording medium, or may be distributed through a communication network and taken into the CPU.

  In addition, the present invention adds a camera shake sensor and a posture sensor to the video camera 150 of the second embodiment, and information on the magnitude of the camera swing caused by camera shake, panning and tilting magnitude information, and A configuration is also included in which a pseudo-stereoscopic image signal of an optimal depth model is generated using roll angle information about the optical axis of the image sensor, and the pseudo-stereoscopic image signal is recorded on a recording medium, as in the third embodiment. Further, the basic depth model type image is not limited to the three-dimensional structure of A, B, and C described above, and may be a plurality of types other than the three types.

100, 150, 170 Video camera 101 Zoom lens 102 Focus lens 103 Diaphragm 104 Image sensor 105 Zoom lens driving unit 106 Focus lens driving unit 107 Aperture driving unit 108, 152, 173 Central processing unit (CPU)
109, 174 Memory 112, 151 Image storage unit 113, 153 Image processing unit 114 Recording / playback I / F unit 115 2D3D conversion unit 118 Operation unit 171 Camera shake sensor 172 Attitude sensor 200 Depth estimation data generation unit 202 High-frequency component evaluation at the top of the screen Unit 203 high-frequency component evaluation unit 204 at the bottom of the screen 204 synthesis ratio determination unit 205 switch 206 depth model synthesis unit 207 to 209 frame memory 210, 301 control signal determination unit 211 weighting unit 212 addition unit 300 stereo pair generation unit 302 texture shift unit 303 Occlusion compensation unit 304 Post processing unit

Claims (7)

The subject is a person from a non-stereoscopic image signal obtained by photoelectrically converting the optical image of the subject imaged on the imaging surface of the imaging device through an optical system including a zoom lens, a focus lens, and an aperture by the imaging device. A face size information acquiring means for acquiring the face size information of the person;
Basic depth model generating means for generating images of a plurality of basic depth model types indicating a basic scene for generating a pseudo-stereoscopic image signal from the non-stereoscopic image signal;
Based on a first control signal indicating a synthesis ratio of the plurality of basic depth model type images, the plurality of basic depth model type images supplied from the basic depth model generating means are combined to generate a depth model image. Depth model synthesis means for generating
Weighting means for multiplying the non-stereo image signal by the weighting coefficient indicated by the second control signal based on a second control signal indicating a weighting coefficient for weighting the non-stereo image signal;
Depth estimation data generation means for generating depth estimation data from the non-stereoscopic image signal weighted by the weighting means and the image of the depth model generated by the depth model synthesis means;
A texture for generating the pseudo stereoscopic image signal by shifting the texture of the non-stereo image signal based on the depth estimation data in which the depth and the convergence are adjusted by the third control signal indicating the depth and the congestion. Shifting means;
When calculating and outputting the first to third control signals, respectively, at least one of the first to third control signals to be calculated using the face size information A pseudo-stereoscopic image generation apparatus comprising: a control signal calculation unit that varies the value of. Lens position acquisition means for acquiring respective positions of a zoom lens and a focus lens constituting the optical system;
Subject estimated distance calculation means for calculating an estimated distance from the position of the focus lens to the subject based on the zoom lens position and the position of the focus lens acquired by the lens position acquisition means;
A scene serving as a basis for generating a pseudo-stereoscopic image signal from a non-stereoscopic image signal obtained by photoelectrically converting an optical image of a subject formed on the imaging surface of the imaging element through the optical system by the imaging element. A basic depth model generating means for generating images of a plurality of basic depth model types shown;
Based on a first control signal indicating a synthesis ratio of the plurality of basic depth model type images, the plurality of basic depth model type images supplied from the basic depth model generating means are combined to generate a depth model image. Depth model synthesis means for generating
Weighting means for multiplying the non-stereo image signal by the weighting coefficient indicated by the second control signal based on a second control signal indicating a weighting coefficient for weighting the non-stereo image signal;
Depth estimation data generation means for generating depth estimation data from the non-stereoscopic image signal weighted by the weighting means and the image of the depth model generated by the depth model synthesis means;
A texture for generating the pseudo stereoscopic image signal by shifting the texture of the non-stereo image signal based on the depth estimation data in which the depth and the convergence are adjusted by the third control signal indicating the depth and the congestion. Shifting means;
When calculating and outputting each of the first and third control signals, at least one of the first and third control signals to be calculated using the calculated estimated distance information to the subject. A pseudo-stereoscopic image generation device comprising: control signal calculation means for changing the value of any one of the control signals. Optical system information acquisition means for acquiring the position of the zoom lens, the position of the focus lens, and the aperture value of the diaphragm, respectively, constituting the optical system;
A depth-of-field calculating means for calculating an estimated depth of field based on the aperture value and the zoom lens position acquired by the optical system information acquiring means;
A scene serving as a basis for generating a pseudo-stereoscopic image signal from a non-stereoscopic image signal obtained by photoelectrically converting an optical image of a subject formed on the imaging surface of the imaging element through the optical system by the imaging element. A basic depth model generating means for generating images of a plurality of basic depth model types shown;
Based on a first control signal indicating a synthesis ratio of the plurality of basic depth model type images, the plurality of basic depth model type images supplied from the basic depth model generating means are combined to generate a depth model image. Depth model synthesis means for generating
Weighting means for multiplying the non-stereo image signal by the weighting coefficient indicated by the second control signal based on a second control signal indicating a weighting coefficient for weighting the non-stereo image signal;
Depth estimation data generation means for generating depth estimation data from the non-stereoscopic image signal weighted by the weighting means and the image of the depth model generated by the depth model synthesis means;
A texture for generating the pseudo stereoscopic image signal by shifting the texture of the non-stereo image signal based on the depth estimation data in which the depth and the convergence are adjusted by the third control signal indicating the depth and the congestion. Shifting means;
When calculating and outputting each of the first to third control signals, at least any one of the first to third control signals to be calculated using the calculated information on the estimated depth of field. A pseudo-stereoscopic image generation apparatus comprising: control signal calculation means for changing a value of the one control signal. Lens position acquisition means for acquiring respective positions of a zoom lens and a focus lens constituting the optical system;
Among the non-stereo image signals obtained by photoelectrically converting the optical image of the subject imaged on the imaging surface of the imaging element through the optical system by the imaging element, a predetermined small area in the imaging screen of the imaging element Automatic focus adjustment means for moving and controlling the position of the focus lens in order to obtain a focus position at which the high-frequency component of the non-stereoscopic image signal in the ranging area is the maximum,
Position data acquisition means for acquiring position data of the ranging area in the imaging screen;
Subject estimated distance calculating means for calculating an estimated distance from the focus position of the focus lens to the subject;
Basic depth model generating means for generating images of a plurality of basic depth model types indicating a basic scene for generating a pseudo-stereoscopic image signal from the non-stereoscopic image signal;
Based on a first control signal indicating a synthesis ratio of the plurality of basic depth model type images, the plurality of basic depth model type images supplied from the basic depth model generating means are combined to generate a depth model image. Depth model synthesis means for generating
Weighting means for multiplying the non-stereo image signal by the weighting coefficient indicated by the second control signal based on a second control signal indicating a weighting coefficient for weighting the non-stereo image signal;
Depth estimation data generation means for generating depth estimation data from the non-stereoscopic image signal weighted by the weighting means and the image of the depth model generated by the depth model synthesis means;
A texture for generating the pseudo stereoscopic image signal by shifting the texture of the non-stereo image signal based on the depth estimation data in which the depth and the convergence are adjusted by the third control signal indicating the depth and the congestion. Shifting means;
When calculating and outputting the first to third control signals, the first and third control signals to be calculated according to the position data of the ranging area and the estimated distance to the subject. A pseudo-stereoscopic image generation apparatus comprising: control signal calculation means for varying at least one of the control signals. Detects camera shake information consisting of information on the amplitude of the image sensor that obtains non-stereoscopic image signals by photoelectrically converting the optical image of the subject imaged on the imaging surface through the optical system, and information on the size of panning and tilting. Camera shake detection means for
Basic depth model generating means for generating images of a plurality of basic depth model types indicating a scene serving as a basis for generating a pseudo-stereoscopic image signal from the non-stereoscopic image signal obtained by the imaging device;
Based on a first control signal indicating a synthesis ratio of the plurality of basic depth model type images, the plurality of basic depth model type images supplied from the basic depth model generating means are combined to generate a depth model image. Depth model synthesis means for generating
Weighting means for multiplying the non-stereo image signal by the weighting coefficient indicated by the second control signal based on a second control signal indicating a weighting coefficient for weighting the non-stereo image signal;
Depth estimation data generation means for generating depth estimation data from the non-stereoscopic image signal weighted by the weighting means and the image of the depth model generated by the depth model synthesis means;
A texture for generating the pseudo stereoscopic image signal by shifting the texture of the non-stereo image signal based on the depth estimation data in which the depth and the convergence are adjusted by the third control signal indicating the depth and the congestion. Shifting means;
When calculating and outputting each of the first to third control signals, the camera shake information is used to calculate the value of at least one of the first to third control signals to be calculated. A pseudo-stereoscopic image generation apparatus comprising: a variable control signal calculation unit. A roll angle detection means for detecting a roll angle around an optical axis of an image pickup element that photoelectrically converts an optical image of a subject imaged on an imaging surface through an optical system to obtain a non-stereoscopic image signal;
Basic depth model generating means for generating images of a plurality of basic depth model types indicating a scene serving as a basis for generating a pseudo-stereoscopic image signal from the non-stereoscopic image signal obtained by the imaging device;
Based on a first control signal indicating a synthesis ratio of the plurality of basic depth model type images, the plurality of basic depth model type images supplied from the basic depth model generating means are combined to generate a depth model image. Depth model synthesis means for generating
Weighting means for multiplying the non-stereo image signal by the weighting coefficient indicated by the second control signal based on a second control signal indicating a weighting coefficient for weighting the non-stereo image signal;
Depth estimation data generation means for generating depth estimation data from the non-stereoscopic image signal weighted by the weighting means and the image of the depth model generated by the depth model synthesis means;
A texture for generating the pseudo stereoscopic image signal by shifting the texture of the non-stereo image signal based on the depth estimation data in which the depth and the convergence are adjusted by the third control signal indicating the depth and the congestion. Shifting means;
When calculating and outputting each of the first to third control signals, the roll angle detection information detected by the roll angle detection means is used to calculate the first and third control signals. A pseudo-stereoscopic image generation apparatus comprising: control signal calculation means for changing a value of at least one of the control signals. In a camera having an image sensor that photoelectrically converts an optical image of a subject imaged on an imaging surface through an optical system to obtain a non-stereoscopic image signal,
Subject estimated distance calculating means for calculating an estimated distance from the camera to the subject;
Shooting scene information based on a signal change between two adjacent frames of a non-stereoscopic image signal output from an image sensor that photoelectrically converts an optical image of a subject imaged on an imaging surface through the optical system and the estimated distance Shooting scene information acquisition means for acquiring
Basic depth model generation means for generating images of a plurality of basic depth model types indicating a scene serving as a basis for generating a pseudo-stereoscopic image signal from the non-stereoscopic image signal output from the imaging device;
Based on a first control signal indicating a synthesis ratio of the plurality of basic depth model type images, the plurality of basic depth model type images supplied from the basic depth model generating means are combined to generate a depth model image. Depth model synthesis means for generating
Weighting means for multiplying the non-stereo image signal by the weighting coefficient indicated by the second control signal based on a second control signal indicating a weighting coefficient for weighting the non-stereo image signal;
Depth estimation data generation means for generating depth estimation data from the non-stereoscopic image signal weighted by the weighting means and the image of the depth model generated by the depth model synthesis means;
A texture for generating the pseudo stereoscopic image signal by shifting the texture of the non-stereo image signal based on the depth estimation data in which the depth and the convergence are adjusted by the third control signal indicating the depth and the congestion. Shifting means;
Of the first to third control signals to be calculated using the shooting scene information acquired by the shooting scene information acquisition means when calculating and outputting the first to third control signals, respectively. And a control signal calculating means for varying the value of at least one of the control signals.