KR20130128226A - An optical means and image photographing apparatus - Google Patents

Abstract

An image capturing apparatus according to the present embodiment includes an optical means and a data processor which processes image data photographed by the optical means, wherein the optical means comprises: a main lens array including a plurality of main lenses And a photo sensor array having the same number of photo sensors as the number of the main lenses, and a micro lens array disposed between the main lens array and the photo sensor array to obtain light field data of light passing through the main lens array. It includes.

Description

An optical means and image photographing apparatus

This embodiment discloses an optical means and an image photographing apparatus.

Recently, a plenoptic camera capable of refocusing a captured image has been studied.

These plenotic cameras are also called light field systems, and when a picture is taken, the microscopic images taken by the microlens take over the role of the pixels that make up the picture, and the images taken by the microlens are recognized as pixels. Various corrections can be made to the image.

The embodiment aims to propose an image photographing apparatus having an improved structure.

An image capturing apparatus according to the present embodiment includes an optical means and a data processor which processes image data photographed by the optical means, wherein the optical means comprises: a main lens array including a plurality of main lenses And a photo sensor array having the same number of photo sensors as the number of the main lenses, and a micro lens array disposed between the main lens array and the photo sensor array to obtain light field data of light passing through the main lens array. It includes.

According to the present exemplary embodiment, since the parallax at both ends of the image capturing apparatus is large, there is an advantage in that the application of the 3D camera is easy. In addition, there is an advantage that the size can be reduced while improving the characteristics of the direction resolution and spatial resolution.

1 to 3 are views showing the configuration of comparative examples of an image photographing apparatus.
4 is a diagram illustrating a configuration of an image capturing apparatus according to the present embodiment.
5 and 6 show configurations of the micro lens array according to the present embodiment.
7 shows patterns constituting a micro lens per main lens according to the present embodiment;
8 is a view for explaining that each main lens constituting the main lens array according to the present embodiment has a different aperture.
9 is a view for explaining a liquid crystal element compensation method of a misaligned main lens;
FIG. 10 is a view for explaining a compensation method using parallel light irradiation on a misaligned main lens; FIG.

Hereinafter, the present embodiment will be described in detail with reference to the accompanying drawings. It should be understood, however, that the scope of the inventive concept of the present embodiment can be determined from the matters disclosed in the present embodiment, and the spirit of the present invention possessed by the present embodiment is not limited to the embodiments in which addition, Variations.

In the following description, the word " comprising " does not exclude the presence of other elements or steps than those listed.

In order to explain an embodiment of the present invention, various comparative examples are mentioned first.

1 to 3 are diagrams showing the configuration of comparative examples of an image photographing apparatus.

First, as shown in FIG. 1, a case consisting of a single main lens and a single photo sensor may be the first comparative example, and may be a general image capturing apparatus. Since the image capturing apparatus records an image of an image incident in all directions on one photo sensor, there is no direction resolution.

Next, as shown in FIG. 2, the case of the plurality of main lenses 10 and the plurality of photo sensors 20 may be a second comparative example. Such an image photographing apparatus may include an N * N array. Since it includes a main lens and a photo sensor, the direction resolution can be greatly increased to N * N, but the volume increase and manufacturing cost due to N * N's main lens can be increased.

Next, as illustrated in FIG. 3, an image photographing apparatus including a single main lens and a single photo sensor, wherein a micro lens array is configured between the main lens and the photo sensor to obtain light filter data. It can be a comparative example of 3.

In this case, in the case where the microlens array constitutes a microlens array of M * M pixel size, the microlens array may have a direction resolution of M * M, and thus may be implemented as a light field system. However, since one photo sensor is used, the spatial resolution of light passing through the M * M microlens array can be reduced to 1 / (M * M), limiting the use of the light field system. This can be

Based on the understanding of the comparative examples that can be configured in this way, it is possible to more accurately understand the technical features of the image capturing apparatus of the present embodiment.

4 is a diagram illustrating a configuration of an image photographing apparatus according to an exemplary embodiment.

Referring to FIG. 4, the image capturing apparatus of the present exemplary embodiment performs image processing such as refocusing of an image using light field data on the optical means 100 and the image photographed by the optical means 100. It includes a data processing unit 200 to enable.

In detail, the optical means 100 of the image capturing apparatus of the present embodiment includes a main lens array 110 including at least two main lenses and a light field data for obtaining light passing through each of the main lenses. The micro lens array 120 and a plurality of photo sensors 130 for receiving the light passing through the micro lens array 120 and converts the light into an electrical signal.

In the main lens array 110, a plurality of main lenses 110a, 110b, 110c, and 110d are fixed by a lens fixing member such as P * Q, and in the drawing, the main lenses are mounted in a 2 * 2 array. Is shown. The photo sensor 130 is configured with the same number of P * Qs to correspond to the number of main lenses.

In addition, the micro lens array 120 is configured in a pattern corresponding to each of the main lenses, the micro lens array corresponding to each main lens may have a structure in which the micro lenses are arranged in A * B.

The micro lens array 120 may be understood to have sub array patterns corresponding to the respective main lenses. Referring to FIG. 5B, when the main lens is configured of 2 * 2, The micro lens array 120 has four sub array patterns 121, 122, 123, and 124. And, each of the sub array patterns is arranged in the micro lenses A * B.

That is, in the present embodiment, a plurality of main lenses are configured as P * Q, and the array size of microlenses for acquiring light field data with respect to the light passing through each main lens is arranged as A * B. A photo sensor for receiving light passing through the micro lenses and converting the light into an electrical signal is configured of P * Q corresponding to the number of main lenses.

Such a configuration has the following excellent effects compared to the comparative examples described above.

In the first comparative example, there is one main lens, and also one porter sensor. In the second comparative example, since there are a plurality of main lenses and photo sensors, the main lens array is assumed to be N * N, and the photo sensor array is also assumed to be N * N. In the third comparative example, since there is one main lens, one photo sensor, and multiple micro lens arrays (pixel resolution per micro lens), the pixel resolution per micro lens is assumed to be M * M.

In the present embodiment for contrast with these comparative examples, since there are a plurality of main lenses, the same photo sensor is configured for the number of the main lenses, and the micro lens array is also configured, the main lens array is (N / n) *. Assume (N * n), and pixel resolution per micro lens corresponding to each main lens is assumed to be m * m, and the photo sensor array configured as the same number as the main lens array is also (N / n) * (N * n).

In this case, according to the configuration of the present embodiment, the direction resolution is N * N times as compared with the first comparative example, the main lens array is 1 / n ² compared with the second comparative example, and the photo sensor array is also 1 / n ²

In addition, if m and n are the same, the direction resolution may be the same as that of the second comparative example and N * N, while the spatial resolution is 1 / n ² and may have superior characteristics than the third comparative example.

The results of comparing the optical characteristics of the comparative examples and the present embodiment can be summarized as shown in the table below.

First Comparative Example

2nd comparative example
Comparative Example 3
Example
Main Lens Array

One
N * N
One
(N / n) * (N / n)
Per microlens
Pixel resolution

0
0
M * M
m * m
Photo sensor array

One
N * N
One
(N / n) * (N / n)
Direction resolution

0
N * N
M * M
(N / n) ² * m ²
Space resolution

One
One
1 / (M * M)
1 / (m * m)
3D parallax

N / A
Great
Not good
Great
size

littleness

greatness

littleness

Great

On the other hand, the data processing unit 200 receives the image data along with the position information for each photo sensor from the optical means 100. The data processor 200 may interpret the images acquired by the photo sensor 130 and reassemble the images to obtain a desired image effect. Image processing such as view change and refocusing of an image may be performed using an image having at least two magnifications and light field data. For example, the data processor 200 is a function of the incident angle of the detected light on the two-dimensional focal plane specified by the light detected from the photo sensors 130 at different positions and the position of each photo sensor detecting the light. As an example, the view changed image or the refocused image may be generated.

Meanwhile, FIGS. 5 and 6 will be described with reference to various embodiments of a main lens array having a plurality of main lenses of the present embodiment and a micro lens array arranged to acquire light field data for light passing through the main lenses. See with reference.

5 and 6 are diagrams showing configurations of the micro lens array according to the present embodiment.

First, referring to FIG. 5, a main lens array including main lenses of P * Q is illustrated in FIG. 5A, and micro lens patterns (sub array patterns) 121, 122, 123, and 124 corresponding to each main lens are illustrated. An example consisting of micro lenses of A * B is shown in Fig. 5 (b).

According to an embodiment, a plurality of main lenses are configured, and light filter data of light passing through the respective main lenses is obtained by subarray patterns in which a plurality of microlenses are arranged in A * B. The sensor is transferred to photo sensors corresponding to the main lens.

The main lenses are not necessarily configured to be N * N symmetric, and are shown as P * Q, and A * B because each sub-array pattern of the micro lens array is also not necessarily symmetric. Is shown. However, for each main lens, the number of micro lenses needs to drop to an integer or the number of pixels for one micro lens needs to correspond to an integer.

When P is larger than Q in the array of the main lens, the horizontal resolution of the image capturing apparatus is increased, and when the image capturing apparatus is used in a mobile device such as a smartphone, an aspect ratio having a different aspect ratio from the horizontal length and the vertical length due to the characteristics of the mobile device Since it is advantageous to have, the array of main lenses can be configured such that P> Q.

In an embodiment of the present invention, the minimum number of main lenses is two, and 2 * 2 main lenses and 2 * 2 micro lenses may be configured in sub array patterns corresponding to the respective main lenses ( 5 (b)). It is also possible for each sub-array pattern of the micro lens array to consist of 2 * 2 micro lenses in a 4 * 4 main lens array.

As another example, as shown in FIG. 6, when two main lenses are configured to constitute a 1 * 2 main lens array ((a) of FIG. 6), the micro lens array may include a first sub array pattern ( 125 and the second sub-array pattern 126, each sub-array pattern may be made of 2 * 1 (Fig. 6 (b)) or 1 * 2 (Fig. 6 (c)).

Here, with respect to the main lens array and the arrangement of the micro lenses for each main lens, when A is larger than P, the spatial imaging power will be increased, and conversely, if P is larger than A, It will be a video recording device with a higher resolution.

Although the number of main lens arrays and microlenses per main lens is different from each other, the size of each direction may be the same based on the effective element. Since the photo sensor has a multi-divided structure according to the number of main lenses, the decrease in the direction resolution may reduce the decrease in the direction resolution by increasing the number of pixels corresponding to one microlens by applying a high resolution sensor. .

As a result, in the image capturing apparatus according to the embodiment of the present invention, although the parallax may be smaller than that of the second comparative example, the parallax at both ends of the image capturing apparatus is larger than that of the first and third comparative examples. This has the advantage of easy application of 3D camera. In addition, a camera application in which the micro lens array and the color filter are integrated is possible, and the shapes of the main lens array, the micro lens array, and the photo sensor array may be configured in a rectangular shape as shown, and may be formed in various shapes such as a circle or a hexagon. It may be changed. In addition, each of the main lens and the microlens may have various combinations of size of the outer diameter and the effective diameter.

On the other hand, the configuration of the micro lens array and the color filter may be implemented in various examples, and the configuration of the color filter in the micro lens array has the same effect as selecting different information areas from the main lenses of the main lens array to obtain information of the entire area. Have

7 is a view showing patterns constituting a micro lens per main lens according to the present embodiment.

The microlens patterns illustrated in FIGS. 7A to 7D correspond to subarray patterns for obtaining light field data per microlens. That is, each of FIGS. 7A to 7D corresponds to a sub array pattern constituting the micro lens array, and each sub array pattern is 2 * 2.

That is, in order to obtain light field data with respect to light passing through each main lens, a sub array pattern corresponding to each main lens may be configured as a 2 * 2 pattern. In this case, each R / G / B / IR filter corresponds to a sensor pixel of a subgroup, and these pixels may be formed by 4 * 4 in each pattern.

Figure 7 (a) is an example of configuring the R / G / B filter, (b) is an example of configuring the R / G / B and Y filter, (c) is configured an R / G / B and IR filter (D) is an example configured to partition each of the R / G / B / Y filters. Such micro lens arrays can be fabricated on semiconductor chips to include sensor pixels.

8 is a view for explaining that each of the main lenses of the main lens array according to the present embodiment has different apertures.

According to the present exemplary embodiment, at least two main lenses of the main lens array are configured as described above, and it is also possible to set different sensitivity to receive light by differently configuring the aperture of each main lens. .

For example, as shown in FIG. 8, the first main lens 110a is configured to have an aperture of F4.0 by tightening the aperture, and the second main lens 110b is configured to have an aperture of F2.8. When the incident angle of the light is changed, the micron lens array xdh can also be configured such that its size and density are different. Apertures will be possible in the micro lens array as well as in the main lens.

Hereinafter, a description will be given of a method for individually correcting the center error when a misalignment occurs in assembling the main lens array, the micro lens array and the photo sensor array.

9 is a view for explaining a liquid crystal element compensation method of a misaligned main lens, and FIG. 10 is a view for explaining a compensation method using a parallel light irradiation on a misaligned main lens.

First, referring to FIG. 9, the case of FIG. 9 (a) is an alignment between the main lens, the micro lens array, and the photo sensor. In FIG. 9 (b), the main lens is assembled by shifting a predetermined distance from the center of the photo sensor. It may be the case.

In this case, the lens holder for supporting the main lens may be shifted by a misaligned distance, or the main lens and the lens holder may be tilted by an angle to compensate for the optical axis to be positioned at the center of the photo sensor.

Referring to FIG. 10, a method of correcting an optical axis by irradiating parallel light toward the main lens, FIG. 10 (a) is a light intensity graph when the optical axis coincides with the center of the photo sensor, and FIG. It is a light intensity graph which shows when it misses a predetermined distance from the center of a photo sensor.

When the main lens is assembled by shifting a predetermined distance from the center of the photo sensor as shown in FIG. 10 (b), the center of the sensor and the center of the intensity Y are shifted, and in this case, the peak position of the light intensity in units of pixels of the sensor. Can be found and the center shifted to compensate.

As described above, the image capturing apparatus of the present exemplary embodiment may provide detailed depth information as compared with the camera array of the second comparative example while providing a thinner thickness and similar performance as compared to the plenomic camera like the third comparative example. In addition, it is possible to implement HDR by different exposures for each main lens, and has an advantage in implementation speed.

Equation 1 is an equation for obtaining a refocused image in a plenoptic camera. By shifting and adding, a sub-aperture image of the virtual focal length F 'can be obtained. And this can be extended to the main lens array to obtain the shifted and final final focus image.

To obtain a refocused image using a microlens, an operation amount of O (k ⁴ ) is basically required (k is the number of samples in each dimension in 4D). Since the third comparative example has a large number of samples, a large amount of calculation is required to obtain a refocused image.

However, according to the present exemplary embodiment, the k value is smaller than that of the comparative examples, and the first refocus image is calculated by the micro lens array of the main lens unit, and all the main lenses collect the first refocus image to collect the second refocus image. It is possible to perform faster because the stepwise operation method that finally calculates can be applied.

In addition, in the present embodiment, since the main lenses of the main lens array are designed to have a sub-pixel distance relationship, an image may be obtained for each main lens unit and the size of the image may be enlarged by using a super-resolution operation. . This method can be handled by computing the sub-pixel distance relationship in a super-resolution sized grid and interpolating to the median in the distorted positional relationship. At this time, it is necessary to take this into account because different sub-pixel distances may occur due to the object. The edge component information can also be used to compensate for the edges being damaged in the result.

Claims (6)

An image photographing apparatus comprising an optical means and a data processing unit for processing image data photographed by the optical means,
The optical means,
A main lens array comprising a plurality of main lenses,
A photo sensor array having the same number of photo sensors as the number of main lenses;
And a microlens array disposed between the main lens array and the photo sensor array to obtain light field data of light passing through the main lens array. The method of claim 1,
The main lens array is a structure in which the main lenses are arranged in the number of P * Q,
And the micro lens array comprises sub-array patterns equal to the number of main lenses. 3. The method of claim 2,
And a sub array pattern of the micro lens array is a structure in which micro lenses are arranged in an A * B number. The method of claim 1,
And some of the main lenses constituting the main lens array have different aperture values. A main lens array composed of at least two main lenses;
A micro lens array for obtaining light field data for light passing through the main lens array; And
And a photo sensor array for converting light passing through the micro lens array into an electrical signal.
The photo sensor constituting the photo sensor array is composed of the same number as the number of the main lens,
And the micro lens array comprises a plurality of sub array patterns corresponding to each of the main lenses. The method of claim 5, wherein
And the number of micro lenses formed in the sub array pattern is an integer multiple of the number of main lenses.

Images