KR101683634B1 - Digital imaging apparatus with plenoptic camera scheme - Google Patents

Abstract

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a digital imaging apparatus using a principle of a planop camera, and more particularly, to a digital imaging apparatus using a principle of a planop camera that reflects an image recognition result.
It is an object of the present invention to provide an artificial vision algorithm that improves responsiveness to non-ideal environmental conditions and simplifies a calculation method as optical field imaging through a plenoptic camera.

Description

[0001] DIGITAL IMAGING APPARATUS WITH PLENOPTIC CAMERA SCHEME [0002]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a digital imaging apparatus using a principle of a planop camera, and more particularly, to a technique for refining a refocusing using a principle of a planop camera to minimize a spatial restriction at the time of photographing.

In applications such as artificial vision (machine vision), the most complex artificial visuals currently being developed due to a variety of adverse conditions, such as rapidly changing lighting conditions, especially dark lighting and blurry environments, smoke, fog, glare and underwater conditions It may not be possible to obtain a proper image recognition result even if an algorithm is used.

Even under ideal conditions, complex and unstructured environments, such as straight edges, non-edge or soft objects, can interfere with the operation of artificial vision algorithms developed for the general environment.

Although a general camera is a necessary device for processing artificial vision, artificial vision algorithm in a non-ideal environment condition is very complicated and a technique for simplifying the algorithm is needed.

Recently, a new concept camera called a plenoptic camera is being studied. Plenoptik cameras are often referred to as light field cameras and capture 4D optical field information for a scene using a microlens array (typically a lenticular lens array) or a light coded mask.

The plenoptic camera has a microlens array between the camera lens and the image sensor, re-focuses the light on the image sensor so that the microlens array produces a number of images from a slightly different point in time, Thereby providing an image in which the focus or viewpoint is adjusted.

An example of such a technology using a plenoptic camera is described in Korean Patent No. 10-1483714 entitled " Digital Imaging Apparatus and Method ".

The prior art is directed to a digital imaging device capable of overcoming the problem of a decrease in spatial resolution caused in the formation of optical field data for image re-focusing. The prior art is a technique for enhancing the formation and processing efficiency of optical filter data by combining a coding technique used in communication technology such as CDMA with a plenoptic camera technology.

However, according to the above prior art, the result of the refocusing using the optical field data is limited by the position of the microlens array, the image sensor, and the like, and therefore, the refocusing performance is also limited.

Korean Registered Patent No. 10-1483714 (Registration date 2015.01.12)

An object of the present invention is to propose a digital imaging device capable of adjusting a depth of field (DOF) based on an arbitrary point by refocusing an image already measured using an artificial vision algorithm.

It is another object of the present invention to provide an artificial vision algorithm that improves responsiveness to non-ideal environmental conditions and simplifies a calculation method as optical field imaging through a plenoptic camera.

The artificial vision algorithm proposed by the present invention is intended to be implemented using a complete 4D optical field that measures the amount of light traveling along each ray traversing the sensor.

An object of the present invention is to provide a more refocusing result which is free from the spatial limitation of an optical system in a re-focusing process by providing additional information using an image recognition algorithm in the re-focusing process of an optical field camera.

The present invention is not limited to the position on the optical system in order to obtain the depth of field (DOF) value required as a result of the refocusing, while keeping the physical aperture width of the main lens within a permissible range in reality It is aimed to search for an arbitrary position.

According to an aspect of the present invention, there is provided a digital imaging apparatus including an image pickup unit for obtaining light field data for refocusing a captured image of an object, And a processor for obtaining a refocused image of the object, the processor including an image recognition unit for applying an image recognition algorithm to the captured image, and a processor for processing the obtained optical field data by reflecting a result of applying the image recognition algorithm And a refocusing unit for refocusing the captured image.

Wherein the imaging unit includes a main lens, a photosensor array for capturing rays of light, and a microlens array positioned between the main lens and the photosensor array, the microlens array physically transferring the rays from the main lens And is directed to the photosensor array through a microlens array.

The re-focusing unit adjusts a re-focusing parameter for refocusing the captured image, if the similarity between the captured image and the reference image is less than a preset reference value, based on a result of applying the image recognition algorithm, And adjusting a re-focusing parameter for refocusing the captured image if the deterioration factor of the image displayed on the captured image is equal to or greater than a preset reference value, based on a result of applying the image recognition algorithm, And re-focusing the captured image.

Further, the captured image is refocused using an optical field calculation based on the obtained optical field data.

Wherein the image recognition unit re-applies the image recognition algorithm to the re-focussed image, wherein the re-focussing unit determines whether the image to which the image recognition algorithm is applied again satisfies a predetermined re-focussing criterion, And the image recognition algorithm determines whether to re-focus the re-applied image again.

Also, the re-focusing section may use a light field operation to refocus the captured image to adjust the re-focussing parameters such that the composite aperture plane and the composite sensor plane are positioned independently of the physical main lens and the position of the physical microlens array And refocusing the captured image by adjusting a refocusing parameter such that the synthetic aperture plane and the synthesized sensor plane are positioned between the physical main lens and the microlens array.

In this case, the captured image is refocused by adjusting the refocusing parameters so that the distance between the synthetic aperture plane and the synthesized sensor plane is equal to or greater than the lower limit reference value.

A digital imaging method according to an embodiment of the present invention includes acquiring optical field data for refocusing a captured image of an object, processing the obtained optical field data to obtain a refocused image of the object Applying the image recognition algorithm to the captured image, and refocusing the captured image by processing the obtained optical field data reflecting the result of applying the image recognition algorithm.

Wherein the refocusing step adjusts a refocusing parameter for refocusing the captured image when the degree of similarity between the captured image and the reference image is less than a preset reference value based on a result of applying the image recognition algorithm, And adjusting the re-focusing parameters for re-focusing the captured image when the image deterioration factors appearing on the captured image are equal to or greater than a preset reference value, based on the result of applying the image recognition algorithm. And refocusing the captured image.

The refocusing may further include refocusing the captured image using an optical field operation based on the obtained optical field data.

The method of claim 1, further comprising: re-applying an image recognition algorithm to the refocused image, wherein the refocusing step determines whether the image re-applied to the image recognition algorithm meets a predetermined refocusing criterion, And based on the result, the image recognition algorithm determines whether to re-focus the re-applied image again.

Wherein the refocusing is performed using an optical field operation to refocus the captured image so that the composite aperture plane and the composite sensor plane are positioned independent of the position of the physical main lens and the physical microlens array, And refocusing the captured image by adjusting the refocusing parameters so that the synthetic aperture plane and the synthesized sensor plane are positioned between the physical main lens and the microlens array.

The refocusing may further include refocusing the captured image by adjusting the refocusing parameter such that a distance between a synthetic aperture plane and a synthesized sensor plane is equal to or greater than a lower limit reference value.

According to the present invention, it is possible to implement a digital imaging device capable of adjusting the depth of field (DOF) based on any point by refocusing an image already measured using an artificial vision algorithm.

In addition, according to the present invention, as an optical field imaging through a plenoptic camera using an artificial vision algorithm, it is possible to improve responsiveness to non-ideal environmental conditions and to simplify a calculation method. At this time, the artificial vision algorithm proposed by the present invention can be implemented by using a complete 4D optical field for measuring the amount of light traveling along each ray traversing the sensor.

According to the present invention, by providing the additional information using the image recognition algorithm in the refocusing process of the optical field camera, it is possible to obtain a higher degree of freedom of refocusing out of the spatial restriction of the optical system in the refocusing process.

According to the present invention, while maintaining the physical aperture width of the main lens within a permissible range in reality, it is not limited to a position on the optical system in order to obtain a Depth of Field (DOF) value required as a result of refocusing You can get depth of field values based on an arbitrary position that does not exist. That is, a digital image having a desired depth of field can be obtained by assuming the position of an image sensor plane as a virtual synthetic sensor plane and obtaining a depth of field depth based on a virtual synthesized sensor plane position. According to the present invention, an image recognition algorithm can be used to search for a virtual synthetic sensor plane position, and a virtual synthetic sensor plane position can be additionally adjusted by a re-focusing algorithm according to a determination criterion of an image recognition algorithm have.

1 is a diagram illustrating optical field parameters of two parallel planes separated from each other to illustrate the concept of optical field parameters.
2 is a view showing a structure of a general optical field plenoptic camera.
3 is a diagram illustrating an artificial visual processing system using an optical field according to an embodiment of the present invention.
4 is a diagram illustrating a conceptual model for composite image formation according to an embodiment of the present invention.
5 is a conceptual view of a digital imaging device according to an embodiment of the present invention.
FIG. 6 is a detailed view of an embodiment of the imaging unit of FIG. 5;
7 is an operational flowchart showing a digital imaging method according to an embodiment of the present invention.
8 is a flowchart illustrating a digital imaging method for improving re-focusing efficiency using an image recognition algorithm according to an embodiment of the present invention.
9 is a flowchart illustrating a digital imaging method for determining whether an image quality deterioration factor appears on an image captured using an image recognition algorithm according to an embodiment of the present invention.
10 is an operational flowchart illustrating a digital imaging method of repeating a re-focusing process according to whether or not a re-focusing criterion according to an embodiment of the present invention is satisfied.
11 is a diagram illustrating an optical field plenoptic camera module applied to a terminal according to an exemplary embodiment of the present invention.
12 is a detailed view of the optical field plenoptic camera module applied to the terminal of FIG.

Other objects and features of the present invention will become apparent from the following description of embodiments with reference to the accompanying drawings.

Preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the following description of the present invention, a detailed description of known functions and configurations incorporated herein will be omitted when it may make the subject matter of the present invention rather unclear.

However, the present invention is not limited to or limited by the embodiments. Like reference symbols in the drawings denote like elements.

1 is a diagram illustrating optical field parameters of two parallel planes separated from each other to illustrate the concept of optical field parameters. One embodiment of the present invention may employ the configuration of FIG. 1 to implement all or part of the configuration of the present invention.

A general camera, not a light field camera, does not record most of the light distribution input from the outside. The loss information generated at this time is obtained not by obtaining only a two-dimensional image of the total amount of light at each point of the photosensor but by measuring 4D optical field data measuring the amount of light moving along each ray crossing the sensor , And can be captured again. In general, a light distribution having a direction reaching each position on the sensor can be recognized as optical field data.

Figure 1 shows two parallel planes and light-field parameters separated by a distance D, where L (u, v, s, t) u, v) of the light beam. Oriented lines can be defined by connecting points on the uv plane with points on the st plane. A system using two planes can describe a ray in terms of position and direction. One of the advantages of the system using these two planes is that the distance D can be selected equal to the focal length of the camera.

2 is a view showing a structure of a general optical field plenoptic camera.

A typical optical field plenoptic camera includes a main lens, a microlens array, and an image sensor.

The main lens may move along the optical axis to match the focus of the subject of interest to a desired depth.

A ray originating at a single point in the object converges to a single point in the focal plane of the microlens array. In the convergence position, the microlenses separate the converged rays based on the directions. As a result, the image of the opening of the main lens is formed by focusing on the pixel array behind the microlens. The image behind the microlens corresponds to the directional resolution in the system depending on the position of the image sensor. In order to maximize the directional resolution, the microlens image is preferably sharp. For this purpose, it is preferable that the focal point of the microlens is formed on the principal plane of the main lens.

3 is a diagram illustrating an artificial visual processing system using an optical field according to an embodiment of the present invention.

The greatest advantage of the optical field artificial vision system of the present invention is that the necessary depth of field (DOF), which is based on an arbitrary position on the image sensor plane, can be maintained while maintaining the physical aperture of the main lens I can get it.

The wide aperture allows more light to be gathered, which makes the light-field data richer so that a relatively clear image can be obtained even if the signal has a low noise ratio.

The decoding unit 320 converts the already stored image sensor data acquired from the optical field camera 310 into optical field data L (u, v, s, t) using a decoding algorithm.

The refocusing unit 340 transforms the optical field data L (u, v, s, t) converted by the decoding unit 320 using the re-focusing algorithm to generate a composite optical field L '(u', v ' s ', t') can be obtained. The meaning of the composite light field L '(u', v ', s', t') will be described later with reference to FIG.

The re-focusing unit 340 generates a virtual synthetic aperture and a synthetic synthetic sensor plane for the light field data L (u, v, s, t) transformed by the decoding unit 320, , A synthetic image having a depth of field (DOF) higher than that of the digital image by the first optical field data L (u, v, s, t) can be obtained by arranging the focal point position on the virtual synthesized sensor plane.

The image recognition unit 360 performs image recognition on the composite image using the image recognition algorithm and transmits the result of the image recognition to the feedback unit 350. [ At this time, the result of the image recognition can determine conditions such as whether the current depth of field (DOF) is satisfactory, whether the current image capturing environment is sufficient to obtain a recognizable image, and the like.

The feedback unit 350 may generate information on whether the conditions of the illumination and the imaging are satisfactory and whether the current depth of field (DOF) needs to be re-adjusted and transmit the information to the refocusing unit 340 or the refocusing unit 330. The conditions of illumination and imaging that the feedback unit 350 can check include, for example, poor lighting, underwater, whether the light is attenuating media ) And the like. Here, smoke, fog, and the like are examples of a medium having a large attenuation.

If the depth of field (DOF) of the currently refocused image is unsatisfactory, the refocusing unit 330 may further re-adjust the depth of field by further applying a refocusing process and an image processing algorithm.

The re-ordering unit 330 may perform an enlargement or reduction operation on the currently reconstructed image using software processing on the reconstructed image. This magnification or reduction operation can be determined and executed based on the depth of field (DOF) or sharpness of the reconstructed image. For example, if the current reconstructed image is sufficiently clear, the re-ordering unit 330 may enlarge the currently reconstructed image to obtain a high resolution version of the image. Or if the current reconstructed image is still not clear despite the re-focusing process, the re-ordering unit 330 may reduce the currently reconstructed image to obtain a lower resolution version of the image.

 The image recognition unit 360 may check non-ideal illumination and non-ideal illumination conditions or images for image conditions, and examples of non-ideal conditions include refocusing the image such as smoke or fog The refocusing unit 340 or the refocusing unit 330 may utilize this through the feedback unit 350. FIG.

4 is a diagram illustrating a conceptual model for composite image formation according to an embodiment of the present invention.

Referring to FIG. 4, there is shown a concept of a synthetic camera that is positioned and refocused differently from the image actually obtained. The synthetic camera introduced here is a virtual camera, and its virtual configuration is the same as a conventional camera. That is, the composite camera will have a synthetic aperture and a synthetic sensor plane. The synthetic sensor plane may be assumed to be a synthetic film plane depending on the embodiment.

The composite camera can be modeled using the following four parameters. The four parameters are the aperture size, the aperture location, the depth of lens, and the depth of the sensor plane.

4 shows an embodiment in which the synthetic aperture (lens) of the synthetic camera is parallel to the synthesized sensor plane, but the spirit of the present invention is not limited to this, and can also be applied to a case in which the synthesized aperture and synthetic sensor plane are not parallel. If the composite aperture and the synthesized sensor plane are not parallel, the ray tracing technique may be applied to track the optical field based on the acquired optical field data.

는 합성 센서 평면 (s', t')과 합성 개구 평면(u', v') 사이를 이동하는 빛(광 필드 데이터)를 나타낸 것이다. 이에 따라, 합성 센서 평면에 나타난 조도 화상 값은 수학식 1로 나타낼 수 있다.Composite light field data

(Light field data) moving between the synthetic sensor plane s ', t' and the synthetic aperture plane u ', v'. Accordingly, the illuminance image value displayed on the synthesized sensor plane can be expressed by Equation (1).

[Equation 1]

The relation between the optical field L actually obtained and the synthetic optical field L 'can be expressed by a coordinate transformation formula as shown in the following equation (2).

&Quot; (2) "

Combining Equations (1) and (2), the illuminance image value of Equation (1) can be rewritten as Equation (3).

&Quot; (3) "

는 합성 카메라의 합성 개구의 면적 값을 나타내며, 수학식 3을 이용하여 스테레오 비전을 포함한 합성 카메라의 이미지를 연산할 수 있다.At this time,

Represents the area value of the synthetic aperture of the synthetic camera, and the image of the synthetic camera including the stereo vision can be calculated using Equation (3).

를 조정하여 최종적으로 원하는 리포커싱 영상, 즉, 합성 영상을 얻을 수 있다.Referring again to FIG. 3, the refocusing unit 340 may generate a refocused composite image by moving the synthesized sensor plane including the stereo vision to an arbitrary position. In the present invention, the re-smoothing unit 330 may perform a re-focusing operation based on the image recognition result of the image recognizing unit 360 and the environmental condition determination result of the feedback unit 350, Lt; / RTI > can perform additional refocusing algorithms as described above. At this time, in the refocusing algorithm performed in the refocusing unit 340 or the refocusing unit 330,

And finally, a desired refocusing image, that is, a composite image can be obtained.

Narrowing the synthetic aperture can improve depth of field (DOF). The refocusing unit 340 or the re-sharpening unit 330 may not be able to detect the sharpness of the image itself if the conditions of illumination or image pickup are poor or the sharpness of the image itself is low as a result of the judgment by the image recognizing unit 340 or the feedback unit 350 Refocusing can be performed in a direction in which the synthetic aperture is further narrowed to increase the depth of field.

The macroscopic effect of a microscopic object can be obtained by the same principle and for an object with a soft edge or an edge, ) Can be improved.

5 is a detailed view of a digital imaging device 500 according to an embodiment of the present invention.

The digital imaging device 500, which adds an image recognition process to the existing optical field camera technique of the present invention and adjusts a re-focusing parameter (a concept including position adjustment such as a synthetic sensing plane) by reflecting an image recognition result, And a processor 520. The processor 520 includes an image recognition unit 521 and a re-focusing unit 522. The image recognition unit 521 and the re-

The imaging unit 510 acquires the light field data for refocusing the captured image of the object. In this case, the object means an object or a scene, and the captured image means an image recognized by a pixel value of an actual physical sensor.

The processor 520 processes the acquired optical field data to obtain a refocused image of the object.

The image recognition unit 521 applies an image recognition algorithm to the captured image, and the refocusing unit 522 processes the acquired optical field data reflecting the result of applying the image recognition algorithm to refocus the captured image have.

At this time, the captured image may be refocused using an optical field operation based on the obtained optical field data.

In addition, the image recognition algorithm may use an image recognition algorithm using quantum computing to solve a complicated calculation process of image recognition. For example, a point of interest of a captured image is set, a relation vector between points of interest is generated, And the interaction between the vectors can be projected to the interaction between the dipoles in the physics model (Ising model). Accordingly, complex problems of image recognition can easily be solved by using quantum computing hardware based on the eigenzone model.

In this case, it is possible to use adiabatic quantum computing (AQC) technique which obtains a solution in a target state by causing an adiabatic change of Hamiltonian among quantum computing hardware using an eigenmodeling model. In particular, among trapping ions, Based spin-phonon coupling and solve the complex computation process of image recognition based on a physical model that can solve quantum computing hardware.

In an exemplary embodiment of the present invention, the re-focusing unit 522 may be configured such that when the similarity between the captured image and the reference image is less than a preset reference value, that is, It is possible to refocus the captured image by adjusting the re-focusing parameters in order to refocus the captured image when the focus of the captured image is blurry or the captured image is blurry.

At this time, the re-focusing parameter is adjusted to increase the depth of field (DOF). Depth of field means the extent to which a picture in photography is perceived as being in focus.

According to another embodiment of the present invention, the re-focusing unit 522 may perform the re-focusing for re-focusing the captured image when the image deterioration factor appearing in the captured image is equal to or greater than a preset reference value, You can adjust the parameters to refocus the captured image.

In this case, the image deterioration factor may include image distortion, poor lighting, underwater environment, fog, smoke, and the like. When there is a deterioration factor of image quality, the refocusing parameter is the depth of field (DOF) Adjust in the increasing direction.

According to another embodiment of the present invention, the re-focused image may be re-image-recognized. In this case, when the result of the image recognition is less than a predetermined reference value, that is, To adjust the depth of field DOF again to increase the depth of field.

In this case, the image recognition unit 521 re-applies the image recognition algorithm to the refocused image, and the re-focusing unit 522 determines whether the image to which the image recognition algorithm is applied again satisfies a preset re- Based on the result, the image recognition algorithm can decide whether to re-focus the re-applied image again.

In another embodiment, the user may select a desired portion of the captured image. Accordingly, the user's selection is input, and the user recognizes the focus of the selected portion again to apply the image recognition algorithm, Determines whether the reapplied image meets a predetermined refocusing criterion, and based on the determination result, the image recognition algorithm may decide to re-focus the reapplied image again.

In one embodiment of the present invention, the refocusing portion 522 may determine the position of the composite aperture plane and the composite sensor plane.

For example, optical field calculations can be used to determine whether a synthetic aperture plane and a synthetic film / sensor plane are located at any position, regardless of the position of the physical main lens and physical microlens array, The captured image can be refocused by adjusting the re-focusing parameters.

Another embodiment may refocus the captured image by adjusting the refocusing parameters such that the composite aperture plane and the composite sensor plane are positioned between the physical main lens and the microlens array.

At this time, it is also possible to refocus the captured image by adjusting the refocusing parameter so that the synthetic aperture plane and the synthetic sensor plane do not coincide or overlap with each other by setting a lower limit reference value in the distance between the synthetic aperture plane and the synthesized sensor plane.

6 is a detailed view of an imaging unit according to an embodiment of the present invention.

The imaging unit 510 included in the digital imaging apparatus 500 of the present invention includes a main lens 511, a photosensor array 513, and a microlens array 512.

At this time, the microlens array 512 and the photosensor array 513 implement an image sensor. By using the digital imaging device 500, it is possible to acquire a refocused image or an image viewed from various angles (i.e., adjust the view of an image).

The rays from a single point on the object (object) 514 in the imaged scene reach a single convergent point on the focal plane of the microlens array 512. [ The microlens 512 at this convergence point separates these light beams based on the direction of light, and produces a focused image of the aperture of the main lens 511 on the photosensor below the microlens.

The photosensor array 513 detects the light incident thereon and produces an output that is processed using one or more of the various components. The output optical data is transmitted to the processor 520 using the data together with positional information for each photosensor that provides the data when generating an image of the scene including, for example, the subject 514.

The processor 520 may be implemented, for example, with a computer or other processing circuitry optionally implemented on a common component (e.g., a chip) or on a different component. Depending on the embodiment, a portion of the processor 520 may be implemented within the imaging section 510, while the other portion may be implemented in an external computer. The processor 520 is configured to process the image data and refocus the image of the scene including the subject 514.

The processor 520 may perform refocusing using the detected light or the characteristics of the detected light in conjunction with the known direction in which light has reached the microlens array 512 (calculated using the known location of each photo sensor) ≪ / RTI > selectively refocus and / or correct the data as it forms an image that can be corrected.

The digital imaging device 500 is implemented in various ways depending on the application. For example, although a microlens is shown as an example and some distinguishable microlenses, the array is typically implemented with multiple (e.g., thousands or millions) microlenses. The photosensor array 513 generally has several photosensors for each microlens in the microlens array 512.

The size, or pitch, of each pixel of the photosensor array 513 is relatively finer than the microlens array 512. In addition, the micro-lenses in the microlens array 512 and the photosensors in the photosensor array 513 are generally positioned such that light traveling through the respective microlenses to the photosensor array does not overlap with light traveling through the adjacent microlenses do.

The main lens 511 has the same performance as being moved horizontally along the optical axis to focus on a subject of interest at the desired depth "d" as illustrated between the main lens 511 and the exemplary imaging subject 514 . Thus, it is possible to perform refocusing at each position of interest based on the obtained optical field data.

For example, rays from a single point of the subject 514 are shown for this illustration. These rays reach a single convergence point in the microlens on the focal plane of the microlens array 512. The microlenses are separated based on the direction of these rays to produce focused image and light field data of the aperture of the main lens 511 on the set of pixels in the pixel array below the microlenses.

The optical field data L (u, v, s, t) crosses the main lens 511 at (u, v) s, t), the light traveling along a ray intersecting the plane of the microlens array 512. For example, intensity values that pass through the position (u, v) of each sub-aperture of the main lens 511 in each microlens s, t. Here, the sub-aperture means the number of directional resolutions of the main lens 511. For example, when the number of sub-apertures is 196, each microlens array 512 is composed of 196 pixels.

Each photosensor in the photosensor array 513 may be implemented to provide a value representative of the set of rays directed to the photosensor through the main lens 511 and the microlens array 512. [ That is, each photosensor generates an output in response to light incident on the photosensor, and the position of each photosensor with respect to the microlens array 512 is used to provide direction information about the incident light.

The processor 520 may generate the re-focused image using the optical field data, i.e., L (u, v, s, t). At this time, the processor 520 may determine the direction of light on each photo sensor by using the position of each photo sensor with respect to the microlens. \

The processor 520 may also determine the depth of field of the subject in the scene in which the detected light is emitted and use the depth of field and the direction of the detected light to calculate a composite image that is focused on a different focal plane than the focal plane have.

On the other hand, the image formed below the particular microlens in the microlens array 512 indicates the directional resolution of the system relative to its position on the imaging plane. The main lens 511 is effectively at an optical infinity of the microlens and the photo sensor array 513 is located in one plane at the focal depth of the microlens in order to focus the microlens. The separation distance "s" between the main lens 511 and the microlens array 512 is selected to achieve a sharp image within the field of view of the microlens.

The aperture size of the main lens 511 and the aperture size of the microlenses in the microlens array 512 (e.g., the effective size of the aperture in the lens) are selected to match the particular application in which the digital imaging device 511 is implemented . This method is facilitated by fitting to the f-number (focal ratio: the effective focal length of the lens to the aperture diameter) of the main lens and the microlens.

For example, the present invention can be applied to military equipment, astronomical telescopes, drones, surveillance cameras, CCTV, etc., which require a large lens that does not control the size of the main lens 511.

7 is an operational flowchart showing a digital imaging method according to an embodiment of the present invention.

A digital imaging method of adding an image recognition process to an existing optical field camera technique of the present invention and adjusting a re-focusing parameter (a concept including a position such as a synthetic sensing plane) by reflecting an image recognition result, And acquires optical field data for re-focusing the image (S710). In this case, the object refers to a subject or a scene, and the captured image means an image recognized by a pixel value of an actual physical sensor.

Thereafter, the obtained light field data is processed to acquire a refocused image of the object in step S720, an image recognition algorithm is applied to the captured image in step S730, and the light field obtained by reflecting the result of applying the image recognition algorithm The captured image can be refocused by processing the data (S740).

At this time, the captured image may be refocused using an optical field operation based on the obtained optical field data.

In addition, the image recognition algorithm may use an image recognition algorithm using quantum computing to solve a complicated calculation process of image recognition. For example, a point of interest of a captured image is set, a relation vector between points of interest is generated, And the interaction between vectors can be projected to the interaction (eigen model) between dipoles in the physics model. Accordingly, complex problems of image recognition can easily be solved by using quantum computing hardware based on the eigenzone model.

In this case, it is possible to use an adiabatic quantum computing (AQC) technique which obtains a solution in a target state by causing an adiabatic change of Hamiltonian among quantum computing hardware using an eigenmodeling model. In particular, among trapping ions, Based spin-phonon coupling and solve the complex computation process of image recognition based on a physical model that can solve quantum computing hardware.

The position of the synthetic aperture plane and the composite sensor plane may be determined in one embodiment of the present invention.

For example, optical field calculations can be used to determine whether a synthetic aperture plane and a synthetic film / sensor plane are located at any position, regardless of the position of the physical main lens and physical microlens array, The captured image can be refocused by adjusting the re-focusing parameters.

Another embodiment may refocus the captured image by adjusting the refocusing parameters such that the composite aperture plane and the composite sensor plane are positioned between the physical main lens and the microlens array.

At this time, it is also possible to refocus the captured image by adjusting the refocusing parameter so that the synthetic aperture plane and the synthetic sensor plane do not coincide or overlap with each other by setting a lower limit reference value in the distance between the synthetic aperture plane and the synthesized sensor plane.

8 is a flowchart illustrating a digital imaging method in which a similarity between a captured image and a reference image is compared according to an exemplary embodiment of the present invention.

In an embodiment of the present invention, after the image recognition algorithm is applied to the captured image in the digital imaging method described in FIG. 7 (S830), the similarity between the captured image and the reference image is calculated based on the result of applying the image recognition algorithm (S840). If the similarity is less than a preset reference value, that is, if the captured image is blurred due to recognition by the human eye or if the captured image by the surrounding environment appears blurry, The refocusing parameter may be adjusted to re-focus the captured image, and the captured image may be refocused (S850).

At this time, the re-focusing parameter is adjusted to increase the depth of field (DOF). Depth of field means the extent to which a picture in photography is perceived as being in focus.

FIG. 9 is a flowchart illustrating a digital imaging method for determining whether an image quality deterioration factor appears in a captured image according to an exemplary embodiment of the present invention.

In yet another embodiment of the present invention, after the image recognition algorithm is applied to the captured image described with reference to FIG. 7 (S930), it is determined whether the image quality deterioration factor appearing in the captured image is equal to or greater than a preset reference value ), And if the image deterioration factor is equal to or greater than a preset reference value, the captured image can be refocused by adjusting the re-focusing parameter for refocusing the captured image.

At this time, the image deterioration factor is a factor including image distortion, poor lighting, underwater environment, fog, smoke and the like, and the re-focusing parameter is adjusted to increase the depth of field (DOF) .

10 is an operational flowchart illustrating a digital imaging method for determining whether or not a refocusing criterion is met according to an embodiment of the present invention.

In another embodiment of the present invention, the re-focussed image may be recognized again. In this case, the image recognition algorithm is applied again to the once-refocused captured image described in Fig. 7 (S1030) It is determined whether the result of applying the algorithm satisfies the refocusing criterion (S1040). If the refocusing criterion is not satisfied, i.e., the image is blurry or the image quality is poor, the depth of field (DOF) It is also possible to perform refocusing in step S1050.

In this case, the image recognition algorithm is reapplied to the re-focussed image, and the image recognition algorithm judges whether the re-applied image satisfies the preset re-focusing criteria. Then, based on the determination result, It is possible to decide whether to focus or not.

In addition, the user can select a desired portion of the captured image. Accordingly, the user selects the portion of the user's selected focus, re-recognizes the focus of the selected portion, applies the image recognition algorithm, It is determined whether or not the predetermined re-focusing criteria are met, and based on the determination result, the image recognition algorithm may decide to re-focus the re-applied image again.

The embodiments of the present invention have been described above with reference to the machine vision system of FIG. 3 and the digital imaging device of FIG. The spirit of the invention may be varied in various embodiments. For example, FIG. 11 illustrates an optical field plenoptic camera module applied to a terminal according to an exemplary embodiment of the present invention.

11 is an example of a terminal to which the digital image pickup apparatus 1130 of the present invention is applied. The terminal may be mounted on the rear portion 1110 of the terminal or may be mounted on the inside 1120 of the terminal.

The terminal may be implemented in the form of a smart phone, a mobile communication terminal, a wireless communication terminal, or a portable camera. Although not shown in FIG. 11, according to another embodiment of the present invention, a DOF is adjusted by a handheld camera, a movie camera, a motion picture camera, The present invention can be applied to an apparatus for editing a movie or broadcast image to be used. In one embodiment of the present invention, the digital imaging device 1130 is detachable onto an existing camera lens of the rear portion 1110 of the terminal. Accordingly, the present invention can be applied to a general terminal having a camera lens mounted on the rear surface thereof, and it is possible to refocus an image photographed by the digital imaging device 1130 using image recognition in a terminal.

In another embodiment of the present invention, the digital imaging device 1130 may be mounted in the interior 1120 instead of the camera of the terminal. Accordingly, it is possible to refocus the image photographed by the digital imaging device 1130 using the image recognition in the terminal.

FIG. 12 is a detailed view of an optical field plenoptic camera module applied to the terminal shown in FIG.

The optical field plenoptic camera module 1130 of the present invention includes a lens group 1131, a micro lens array 1132, and an image sensor 1133.

A lens group 1131 may change the focus of the subject of interest to a desired depth along the optical axis.

The light rays of light at a single point of the object converge to a single point on the focal plane of the micro lens array 1132. [ Which separates the light rays of light based on the direction of the focused image in the aperture of the main lens in the pixel array below the microlenses.

Further, an image under the microlens indicates the resolution of the direction of the system with respect to the position in the image sensor 1133, and it is possible through the clear microlens image in order to maximize the direction resolution. Accordingly, the present invention focuses on a technique of refocusing an image captured through an optical field plenoptic camera module 113 using image recognition.

The digital imaging method according to an exemplary embodiment of the present invention may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, and the like, alone or in combination. The program instructions recorded on the medium may be those specially designed and constructed for the present invention or may be available to those skilled in the art of computer software. Examples of computer-readable media include magnetic media such as hard disks, floppy disks and magnetic tape; optical media such as CD-ROMs and DVDs; magnetic media such as floppy disks; Magneto-optical media, and hardware devices specifically configured to store and execute program instructions such as ROM, RAM, flash memory, and the like. Examples of program instructions include machine language code such as those produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

As described above, the present invention has been described with reference to particular embodiments, such as specific elements, and specific embodiments and drawings. However, it should be understood that the present invention is not limited to the above- And various modifications and changes may be made thereto by those skilled in the art to which the present invention pertains.

Accordingly, the spirit of the present invention should not be construed as being limited to the embodiments described, and all of the equivalents or equivalents of the claims, as well as the following claims, belong to the scope of the present invention .

310: Optical Field Camera
320:
330: Reissue Government
340: Re-
350:
360:

Claims (20)

An image pickup section for obtaining light field data for re-focusing the captured image of the object;
Processing the acquired optical field data to obtain a refocused image of the object;
Lt; / RTI >
The processor
An image recognition unit for applying an image recognition algorithm for setting a point of interest on the captured image and modeling a relationship between points of interest; And
A refocusing unit for refocusing the captured image by processing the obtained optical field data reflecting a result of applying the image recognition algorithm,
And a digital camera. The method according to claim 1,
The imaging unit
A main lens;
A photosensor array for capturing light rays; And
A micro lens array positioned between the main lens and the photosensor array;
Lt; / RTI >
Wherein the microlens array directs the rays physically from the main lens to the photosensor array through the microlens array. The method according to claim 1,
The re-
And refocusing the captured image by adjusting a refocusing parameter for refocusing the captured image when the similarity degree between the captured image and the reference image is less than a preset reference value based on a result of applying the image recognition algorithm The digital imaging device The method according to claim 1,
The re-
And refocusing the captured image by adjusting a refocusing parameter for refocusing the captured image when an image quality deterioration factor appearing in the captured image is equal to or greater than a preset reference value based on a result of applying the image recognition algorithm The digital imaging device The method according to claim 1,
The re-
And refocusing the captured image using an optical field calculation based on the obtained optical field data. The method according to claim 1,
The image recognition unit
Applying the image recognition algorithm to the re-focused image again,
The re-
Wherein the image recognition algorithm determines whether the re-applied image satisfies a predetermined re-focusing criterion and determines whether to re-focus the re-applied image based on the determination result, . The method according to claim 1,
The re-
Adjusting the re-focusing parameters to refocus the captured image so that the synthetic aperture plane and the synthesized sensor plane are positioned irrespective of the position of the physical main lens and the physical microlens array using optical field computation. . The method according to claim 1,
The re-
And refocusing the captured image by adjusting a refocusing parameter such that the composite aperture plane and the composite sensor plane are positioned between the physical main lens and the microlens array. An image pickup section for obtaining light field data for re-focusing the captured image of the object;
Processing the acquired optical field data to obtain a refocused image of the object;
Lt; / RTI >
The processor
An image recognition unit for applying an image recognition algorithm to the captured image; And
A refocusing unit for refocusing the captured image by processing the obtained optical field data reflecting a result of applying the image recognition algorithm,
Lt; / RTI >
The re-
And refocusing the captured image by adjusting the refocusing parameter including a depth of field such that the distance between the synthetic aperture plane and the synthesized sensor plane is equal to or greater than a lower limit reference value. The method according to claim 1,
The processor
An image processor for enlarging or reducing the refocused image of the object through software processing of the refocused image of the object,
Further comprising: an image pickup device for picking up the digital image data. Obtaining light field data for refocusing a captured image of an object;
Processing the acquired optical field data to obtain a refocused image of the object;
Applying an image recognition algorithm for setting a point of interest on the captured image and modeling a relationship between points of interest; And
Refocusing the captured image by processing the obtained optical field data reflecting a result of applying the image recognition algorithm;
And outputting the digital image signal. 12. The method of claim 11,
The step of refocusing
And refocusing the captured image by adjusting a refocusing parameter for refocusing the captured image when the similarity degree between the captured image and the reference image is less than a preset reference value based on a result of applying the image recognition algorithm Wherein said digital imaging method comprises the steps of: 12. The method of claim 11,
The step of refocusing
And refocusing the captured image by adjusting a refocusing parameter for refocusing the captured image when an image quality deterioration factor appearing in the captured image is equal to or greater than a preset reference value based on a result of applying the image recognition algorithm Wherein said digital imaging method comprises the steps of: 12. The method of claim 11,
The step of refocusing
And refocusing the captured image using an optical field calculation based on the obtained optical field data. 12. The method of claim 11,
Applying the image recognition algorithm to the refocused image again
Further comprising:
The step of refocusing
Wherein the image recognition algorithm determines whether the re-applied image meets a predetermined re-focusing criterion and determines whether to re-focus the re-applied image based on the determination result, . 12. The method of claim 11,
The step of refocusing
Adjusting the re-focusing parameters to refocus the captured image so that the synthetic aperture plane and the synthesized sensor plane are positioned irrespective of the position of the physical main lens and the physical microlens array using optical field computation. / RTI > 12. The method of claim 11,
The step of refocusing
Refocusing the captured image by adjusting a refocusing parameter such that the composite aperture plane and the composite sensor plane are positioned between the physical main lens and the microlens array. Obtaining light field data for refocusing a captured image of an object;
Processing the acquired optical field data to obtain a refocused image of the object;
Applying an image recognition algorithm to the captured image; And
Refocusing the captured image by processing the obtained optical field data reflecting a result of applying the image recognition algorithm;
Lt; / RTI >
The step of refocusing
And refocusing the captured image by adjusting the refocusing parameter including a Depth of Field such that a distance between a synthetic aperture plane and a synthesized sensor plane is equal to or greater than a lower limit reference value. 12. The method of claim 11,
Enlarging or reducing the refocused image of the object through software processing of the refocused image of the object
Further comprising the steps of: A computer-readable recording medium having recorded thereon a program for executing the method according to any one of claims 11 to 19.

Images