JP2019149674A - Imaging apparatus, imaging method, and image processing apparatus - Google Patents

Abstract

The present invention provides a technique suitable for realizing various functions by reducing calculation processing when performing focus adjustment after imaging and transmitting a sensor image of a lensless imaging apparatus. An imaging apparatus includes: a modulator that modulates light intensity based on a lattice pattern; an image sensor that converts light transmitted through the modulator into an electrical signal to generate a sensor image; and A complex sensor image processing unit that generates a complex sensor image having a complex number; and a data transmission unit that transmits the complex sensor image. [Selection] Figure 1

Description

  The present invention relates to an imaging apparatus, an imaging method, and an image processing apparatus.

  As a background art in this technical field, there is International Publication No. 2017/149687. In this publication, “an imaging device capable of enhancing the functionality of an imaging device by facilitating detection of the incident angle of a light beam that passes through a lattice substrate is obtained. A plurality of pixels arranged in an array on the imaging surface An image sensor that converts an optical image captured into an image signal and outputs the image signal; a modulator that is provided on the light receiving surface of the image sensor and modulates the intensity of light; and an image signal output from the image sensor A first lattice pattern comprising a plurality of concentric circles, and an image storage unit that stores the image signal and a signal processing unit that performs image processing of an image signal output from the image storage unit And the signal processing unit generates a moire fringe image by modulating the image signal output from the image storage unit with a virtual second lattice pattern composed of a plurality of concentric circles, and Depending on the position Can be resolved by the imaging apparatus characterized by changing the size of the concentric grating pattern. It has been described as ".

International Publication No. 2017/149687 International Publication No. 2017/145348

  A lensless image pickup apparatus that does not use a lens is expected as an image pickup apparatus that can achieve a small size and low cost. In addition, network connection of imaging devices has become essential for expanding the application range of imaging devices such as image analysis. However, the above-described Patent Document 1 describes a method for realizing functions such as focus adjustment (refocus), autofocus, and distance measurement when generating a moire fringe image from a sensor image captured by a lensless imaging device. . Patent Document 2 discloses a technique for removing noise by arranging the phase of the front-side grating and the phase of the back-side grating so as to be superposed in all combinations independently of each other. However, when focus adjustment or the like is performed again on an image obtained by imaging, it is necessary to perform calculation processing for development pattern multiplication and noise removal again, and the calculation processing becomes enormous.

  An object of the present invention is to provide a technique suitable for realizing various functions by reducing calculation processing when performing focus adjustment after imaging and transmitting a sensor image of a lensless imaging apparatus. .

  The present application includes a plurality of means for solving at least a part of the above-described problems. Examples of such means are as follows.

  An imaging apparatus according to an aspect of the present invention includes a modulator that modulates the intensity of light based on a lattice pattern, an image sensor that converts the light transmitted through the modulator into an electrical signal, and generates a sensor image; A complex sensor image processing unit that generates a complex sensor image having a complex number from the sensor image; and a data transmission unit that transmits the complex sensor image.

  According to the present invention, it is possible to provide a technique suitable for realizing various functions by transmitting a sensor image of a lensless imaging apparatus by simplifying the processing and reducing the amount of communication data.

  Problems, configurations, and effects other than those described above will become apparent from the following description of embodiments.

It is a figure which shows the structural example of the imaging device which concerns on 1st Embodiment. It is a figure which shows the structural example of an imaging part. It is a figure which shows the other structural example of an imaging part. It is a figure which shows an example of the pattern for imaging | photography. It is a figure which shows the other example of the pattern for imaging | photography. It is a figure explaining that the projection image from the pattern board | substrate surface by an oblique incident parallel light to an image sensor produces in-plane shift | offset | difference. It is a figure which shows an example of the projection image of the pattern for imaging | photography. It is a figure which shows an example of the pattern for development. It is a figure which shows an example of the image developed by a correlation image development system. It is a figure which shows an example of the moire fringe by a moire development system. It is a figure which shows an example of the image developed by the moire development system. It is a figure which shows an example of the combination of the initial phase in a fringe scan. It is a figure which shows an example of the pattern for imaging | photography in a space division | segmentation fringe scan. It is a flowchart which shows the process example of a fringe scan process part. It is a flowchart which shows the process example of the image process part by a correlation image development system. It is a flowchart which shows the process example of the image process part by a moire development system. It is a figure explaining that a pattern for photography is projected when an object exists in infinite distance. It is a figure explaining an imaging pattern being expanded when an object exists in a finite distance. It is a figure which shows the structural example of the imaging device which has a focus function. It is a figure which shows the structural example of the imaging system which concerns on 2nd Embodiment. It is a figure which shows an example of a sensor image. It is a figure which shows an example of the luminance distribution of a sensor image. It is a figure which shows the other example of the luminance distribution of a sensor image. It is a figure which shows an example of the luminance distribution of a complex sensor image. It is a figure which shows an example of the data in the complex space of a complex sensor image. It is a figure which shows an example of the histogram of the real part data of a complex sensor image. It is a figure which shows the other example of the data in the complex space of a complex sensor image. It is a figure which shows the other example of the histogram of the real part data of a complex sensor image. It is a figure which shows the further another example of the data in the complex space of a complex sensor image. It is a figure which shows the further another example of the data in the complex space of a complex sensor image. It is a figure which shows the structural example of the imaging system which concerns on 3rd Embodiment. It is a flowchart which shows the process example of a data amount reduction part. It is a flowchart which shows the other example of a process of the data amount reduction part. It is a flowchart which shows the further another example of the process of a data amount reduction part.

  In the following embodiments, when it is necessary for the sake of convenience, the description will be divided into a plurality of sections or embodiments. However, unless otherwise specified, they are not irrelevant to each other. Some or all of the modifications, details, supplementary explanations, and the like are related.

  In the following embodiments, when referring to the number of elements (including the number, numerical value, quantity, range, etc.), unless otherwise specified, the case is clearly limited to a specific number in principle. It is not limited to the specific number, and may be a specific number or more.

  Further, in the following embodiments, it is needless to say that the constituent elements (including element steps and the like) are not necessarily indispensable unless otherwise specified and clearly considered essential in principle. Yes.

  Similarly, in the following embodiments, when referring to the shape and positional relationship of components and the like, the shape and the like of the component are substantially excluding unless explicitly stated or otherwise considered in principle. It shall include those that are approximate or similar to. The same applies to the above numerical values and ranges.

  In all the drawings for explaining the embodiments, the same members are denoted by the same reference symbols in principle, and the repeated explanation thereof is omitted. Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[First Embodiment]
<Principle of shooting an object at infinity>
FIG. 1 is a diagram illustrating a configuration example of an imaging apparatus 101 according to the first embodiment. The imaging apparatus 101 acquires an image of an object in the outside world without using a lens that forms an image, and includes an imaging unit 102, a fringe scan processing unit 106 (which may be called a complex sensor image processing unit), and an image processing unit. 107 and a controller 108.

  FIG. 2 is a diagram illustrating a configuration example of the imaging unit 102. The imaging unit 102 includes an image sensor 103, a pattern substrate 104, and a shooting pattern 105. The pattern substrate 104 and the imaging pattern 105 can be collectively referred to as a modulator.

  The pattern substrate 104 is fixed in close contact with the light receiving surface of the image sensor 103, and a photographing pattern 105 is formed on the surface of the pattern substrate 104. The pattern substrate 104 is made of a material that is transparent to visible light, such as glass or plastic.

  The photographing pattern 105 is formed by evaporating a metal such as aluminum or chromium by a sputtering method used in a semiconductor process, for example. The shade is given by a pattern in which aluminum is deposited and a pattern in which aluminum is not deposited.

  The formation of the photographing pattern 105 is not limited to vapor deposition, and any method can be used as long as it can realize modulation of transmittance, for example, by adding light and shade by printing with an inkjet printer or the like. Good.

  Here, although visible light has been described as an example, for example, when far-infrared imaging is performed, the pattern substrate 104 is made of a material that is transparent to far-infrared, such as germanium, silicon, chalcogenide, and the like. That is, a material transparent to the wavelength to be imaged can be used for the pattern substrate 104, and a material that blocks the wavelength to be imaged can be used for the image capturing pattern 105.

  Although the method for forming the imaging pattern 105 on the pattern substrate 104 has been described here, the present invention is not limited to this. FIG. 3 is a diagram illustrating another configuration example of the imaging unit 102. The imaging unit 102 can also be realized by a configuration in which the imaging pattern 105 is formed in a thin film and is held by a support member 301 provided instead of the pattern substrate 104. The support member 301 and the photographing pattern 105 can be collectively referred to as a modulator. In the imaging apparatus 101 illustrated in FIG. 1, the shooting angle of view can be changed depending on the thickness of the pattern substrate 104. Therefore, if the support member 301 shown in FIG. 3 has a function capable of changing its length, the shooting angle of view can be changed.

  As shown in FIG. 2 or FIG. 3, the image sensor 103 includes, for example, a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor. On the imaging surface (light receiving surface) of the image sensor 103, pixels 103a as light receiving elements are arranged in an array. The intensity of the light transmitted through the photographing pattern 105 is modulated by the lattice pattern and received by the image sensor 103. The image sensor 103 converts the optical image received by the pixel 103a into an image signal that is an electrical signal and outputs the image signal. The image signal (analog image data) is converted into a digital signal via an analog / digital conversion circuit, for example, and output as digital image data. In this specification, description will be made assuming that the imaging unit 102 outputs image data.

  The fringe scan processing unit 106 performs noise removal by fringe scanning on the image data output from the image sensor 103 and outputs the result to the image processing unit 107. The image processing unit 107 performs predetermined image processing on the image data output from the fringe scan processing unit 106 and outputs the image data to the controller 108. The controller 108 converts the data format of the image data output from the image processing unit 107 as necessary, and stores the data in a storage device (not shown) included in the imaging device 101, or an external host computer or recording medium Or output to

  The controller 108 can be realized by a unit including a processor, a memory, a communication device, a processing circuit, and the like, for example. Further, the controller 108 may be connected to or provided with an input / output interface connected to an external device, such as USB (Universal Serial Bus), HDMI (High-Definition Multimedia Interface), or the like. The fringe scan processing unit 106 and the image processing unit 107 are realized by, for example, an image processing circuit. The fringe scan processing unit 106, the image processing unit 107, and the controller 108 may be integrally configured.

  Next, the imaging principle in the imaging apparatus 101 will be described.

  First, the photographing pattern 105 is a concentric lattice pattern whose pitch is inversely proportional to the radius from the center, and using the radius r and the coefficient β from the reference coordinate which is the center of the concentric circle,

と定義される。撮影用パターン１０５は、この式に比例して透過率が変調されているものとする。

It is defined as It is assumed that the transmittance of the photographing pattern 105 is modulated in proportion to this equation.

  A plate having such stripes is called a Gabor zone plate or a Fresnel zone plate. FIG. 4 is a diagram illustrating an example of the shooting pattern 105, in which a Gabor zone plate of Formula (1) is illustrated. FIG. 5 is a diagram showing another example of the shooting pattern 105, and shows a Fresnel zone plate obtained by binarizing Equation (1) with a threshold value 1.

  Hereafter, for simplification, only the x-axis direction will be described using mathematical formulas. Similarly, it is possible to consider two-dimensional development by considering the y-axis direction.

FIG. 6 is a diagram for explaining that the projected image from the pattern substrate surface to the image sensor due to obliquely incident parallel light causes an in-plane shift. It is assumed that parallel light is incident on the pattern substrate 104 having the thickness d on which the photographing pattern 105 is formed at an angle θ ₀ in the x-axis direction. With geometrical optics, light multiplied by the transmittance of the surface grating is incident on the image sensor 103 with a shift of k = d · tan θ. At this time,

のような強度分布を持つ投影像が画像センサ１０３上で検出される。なお、Φは式（１）の透過率分布の初期位相を示す。この撮影用パターン１０５の投影像（図７）は、式（２）のようにｋシフトして投影される。これが撮像部１０２の出力となる。

A projected image having such an intensity distribution is detected on the image sensor 103. Note that Φ represents the initial phase of the transmittance distribution of the formula (1). The projected image (FIG. 7) of the photographing pattern 105 is projected with k shift as shown in Expression (2). This is the output of the imaging unit 102.

  Next, regarding the processing in the image processing unit 107, development processing by the correlation development method and the moire development method will be described.

  In the correlation development method, a bright point (FIG. 9) with a shift amount k is obtained by calculating a cross-correlation function between the projected image of the photographing pattern 105 (FIG. 7) and the development pattern 801 (FIG. 8). is there. In general, since the amount of calculation increases when the cross-correlation calculation is performed by two-dimensional convolution calculation, the principle of an example of calculation using Fourier transform will be described using mathematical expressions.

  First, the development pattern 801 uses a Gabor zone plate or a Fresnel zone plate as in the case of the shooting pattern 105. Therefore, the development pattern 801 uses the initial phase Φ,

と表せる。現像用パターン８０１は、画像処理部１０７の演算処理内で仮想的なデータとして生成されて使用されるため、式（１）のように１でオフセットさせる必要はなく、負の値を有していても問題ない。

It can be expressed. Since the development pattern 801 is generated and used as virtual data in the arithmetic processing of the image processing unit 107, it does not need to be offset by 1 as in Expression (1), and has a negative value. There is no problem.

  The Fourier transforms of Equation (2) and Equation (3) are respectively

のようになる。ここで、Ｆはフーリエ変換の演算を表し、ｕはｘ方向の周波数座標、括弧を伴うδはデルタ関数である。この式で重要なことはフーリエ変換後の式もまたフレネルゾーンプレートやガボールゾーンプレートとなっている点である。よって、この数式に基づいて、フーリエ変換後の現像用パターン８０１を直接的に生成してもよい。これにより演算量を低減可能である。

become that way. Here, F represents a Fourier transform operation, u is a frequency coordinate in the x direction, and δ with parentheses is a delta function. What is important in this formula is that the formula after Fourier transform is also a Fresnel zone plate or a Gabor zone plate. Therefore, the development pattern 801 after the Fourier transform may be directly generated based on this mathematical expression. Thereby, the amount of calculation can be reduced.

  Next, multiplying Expression (4) and Expression (5),

となる。この指数関数で表された項ｅｘｐ（−ｉｋｕ）が信号成分であり、この項をフーリエ変換すると、

It becomes. The term exp (−iku) expressed by this exponential function is a signal component, and when this term is Fourier transformed,

のように変換され、元のｘ軸においてｋの位置に輝点を得ることができる。この輝点が無限遠の光束を示しており、図１の撮像装置１０１による撮影像にほかならない。

Thus, a bright spot can be obtained at the position of k on the original x-axis. This luminescent spot indicates a light beam at infinity, which is nothing but a photographed image by the imaging device 101 of FIG.

  In the correlation development method, as long as the autocorrelation function of the pattern has a single peak, it may be realized by a pattern not limited to the Fresnel zone plate or the Gabor zone plate, for example, a random pattern.

  Next, in the moiré development system, a moiré fringe (FIG. 10) is generated by multiplying the projected image of the photographic pattern 105 (FIG. 7) and the development pattern 801 (FIG. 8), and the developed image is subjected to Fourier transform. A bright spot (FIG. 11) with a shift amount kβ / π at is obtained. When this moire fringe is shown by a mathematical formula,

となる。この展開式の第３項が信号成分であり、２つのパターンのずれの方向にまっすぐな等間隔の縞模様が、２つのパターンの重なり合った領域一面に作られることがわかる。このような縞と縞の重ね合わせによって相対的に低い空間周波数で生じる縞は、モアレ縞と呼ばれる。この第３項の２次元フーリエ変換は、

It becomes. It can be seen that the third term of this development formula is a signal component, and a striped pattern of equal intervals straight in the direction of deviation between the two patterns is formed over the entire area where the two patterns overlap. Such a fringe generated at a relatively low spatial frequency by overlapping the fringes is called a moire fringe. The two-dimensional Fourier transform of this third term is

のようになる。ここで、Ｆはフーリエ変換の演算を表し、ｕはｘ方向の周波数座標、括弧を伴うδはデルタ関数である。この結果から、モアレ縞の空間周波数スペクトルにおいて、空間周波数のピークがｕ＝±ｋβ／πの位置に生じることがわかる。この輝点が無限遠の光束を示しており、図１の撮像装置１０１による撮影像にほかならない。

become that way. Here, F represents a Fourier transform operation, u is a frequency coordinate in the x direction, and δ with parentheses is a delta function. From this result, it can be seen that a spatial frequency peak occurs at a position of u = ± kβ / π in the spatial frequency spectrum of moire fringes. This luminescent spot indicates a light beam at infinity, which is nothing but a photographed image by the imaging device 101 of FIG.

  In the moire development method, as long as the moire fringes obtained by pattern shifting have a single frequency, the moire development method may be realized by a pattern that is not limited to the Fresnel zone plate or the Gabor zone plate, for example, an elliptical pattern. .

<Noise cancellation>
In the conversion from the expression (6) to the expression (7) and the conversion from the expression (8) to the expression (9), the discussion proceeded with attention to the signal component, but in reality, terms other than the signal component are developed. Inhibit. Therefore, noise cancellation based on fringe scanning is performed.

  For fringe scanning, it is necessary to use a plurality of patterns having different initial phases Φ as the imaging pattern 105. FIG. 12 is a diagram illustrating an example of combinations of initial phases in fringe scanning. Here, when sensor images photographed using four phases satisfying Φ = 0, π / 2, π, and 3π / 2 are calculated according to the following expression, a complex sensor image (complex sensor image) is obtained.

ここで、複素数の現像用パターン８０１は、

Here, the complex development pattern 801 is:

と表せる。現像用パターン８０１は、演算処理内で使用されるため、複素数であっても問題ない。モアレ現像方式の場合、式（１０）と式（１１）を乗算すると、

It can be expressed. Since the development pattern 801 is used in the arithmetic processing, there is no problem even if it is a complex number. In the case of the moire development method, when the equation (10) and the equation (11) are multiplied,

となる。この指数関数で表された項ｅｘｐ（２ｉβｋｘ）が信号成分であり、式（８）のような不要な項が発生せず、ノイズキャンセルされていることが解る。

It becomes. It can be seen that the term exp (2iβkx) represented by the exponential function is a signal component, and an unnecessary term like Expression (8) does not occur and noise cancellation is performed.

  Similarly, confirming the correlation development method, the Fourier transforms of the equations (10) and (11) are respectively

のようになる。次に、式（１３）と式（１４）を乗算すると、

become that way. Next, multiplying Expression (13) and Expression (14),

となる。この指数関数で表された項ｅｘｐ（−ｉｋｕ）が信号成分であり、式（８）のような不要な項が発生せず、ノイズキャンセルされていることが解る。

It becomes. It can be seen that the term exp (-iku) represented by the exponential function is a signal component, and an unnecessary term such as that in Equation (8) does not occur and noise cancellation is performed.

  In the above example, a description has been given using a plurality of patterns of four phases. However, Φ may be set so as to equally divide the angle between 0 and 2π, and is not limited to these four phases. .

  In order to realize the above-described photographing with a plurality of patterns, there are a method of switching patterns by time division (time division fringe scan) and a method of switching patterns by space division (space division fringe scan).

  In order to realize the time-division fringe scan, for example, the photographing pattern 105 is formed by using a liquid crystal display element that can electrically switch and display a plurality of initial phases shown in FIG. 12 (that is, the pattern can be changed). Configure. The imaging unit 102 controls the switching timing of the liquid crystal display element and the shutter timing of the image sensor 103 in synchronization with each other, and the fringe scan processing unit 106 performs a fringe scan calculation after acquiring four images.

  On the other hand, in order to realize the space division fringe scan, for example, as shown in FIG. 13 (a diagram showing an example of the imaging pattern 105 in the space division fringe scan), the imaging pattern 105 having a plurality of initial phases is used. . The imaging unit 102 controls the shutter timing of the image sensor 103, and the fringe scan processing unit 106 acquires one image and then divides the acquired image into four corresponding to each initial phase pattern. Perform a fringe scan operation.

  Next, the fringe scan calculation in the fringe scan processing unit 106 will be described. FIG. 14 is a flowchart illustrating a processing example of the fringe scan processing unit 106.

  First, the fringe scan processing unit 106 obtains a sensor image of a plurality of phase patterns output from the image sensor 103 (one sheet in the case of a spatial division fringe scan and a plurality of sheets in a time division fringe scan). The fringe scan processing unit 106 divides the acquired sensor image for each phase when using the space division fringe scan (1401), and does not perform the process 1401 when using the time division fringe scan. Next, the fringe scan processing unit 106 initializes an output complex sensor image (1402).

  Subsequently, the fringe scan processing unit 106 repeats the processes 1403 to 1405 for each initial phase. For example, in a fringe scan using four phases as shown in FIG. 12, it is repeated four times: Φ = 0, π / 2, π, 3π / 2. The fringe scan processing unit 106 acquires a sensor image of the target initial phase Φ (1403), multiplies exp (iΦ) according to the initial phase Φ (1404), and adds the multiplication result to the complex sensor image ( 1405). The fringe scan processing unit 106 determines whether or not the processing for all initial phases has been completed (1406). If not completed, the processing returns to 1403 (NO in 1406), and if completed, the processing is performed. The process proceeds to 1407 (YES in 1406).

  Finally, the fringe scan processing unit 106 outputs a complex sensor image (1407). The above processing in the fringe scan processing unit 106 corresponds to Expression (10).

  Next, image processing in the image processing unit 107 will be described. FIG. 15 is a flowchart illustrating a processing example of the image processing unit 107 using the correlation development method.

  First, the image processing unit 107 acquires a complex sensor image output from the fringe scan processing unit 106, and performs a two-dimensional fast Fourier transform (FFT) operation on the complex sensor image (1501). Next, the image processing unit 107 generates a development pattern 801 to be used for development processing, multiplies the complex sensor image obtained by the two-dimensional FFT operation (1502), and performs the inverse two-dimensional FFT operation (1503). Since the calculation result is a complex number, the image processing unit 107 converts the absolute value or the real part into a real number and develops (restores) the image to be photographed (1504). Thereafter, the image processing unit 107 performs contrast enhancement processing (1505), color balance adjustment (1506), and the like on the obtained developed image, and outputs it as a captured image. Thus, the image processing of the image processing unit 107 by the correlation development method is completed.

  On the other hand, FIG. 16 is a flowchart showing a processing example of the image processing unit 107 using the moire development method.

  First, the image processing unit 107 acquires a complex sensor image output from the fringe scan processing unit 106. The image processing unit 107 generates a development pattern 801 to be used for development processing, multiplies the complex sensor image (1601), obtains a frequency spectrum by two-dimensional FFT calculation (1602), and a necessary frequency among the frequency spectra is obtained. The area data is cut out (1603). The subsequent processes are the same as the processes 1504 to 1506 in FIG. Thus, the image processing of the image processing unit 107 by the moire development method is completed.

<Principle of photographing a finite distance object>
FIG. 17 is a diagram for explaining that the shooting pattern 105 is projected when the object is at an infinite distance. FIG. 17 shows a state in which the shooting pattern 105 is projected onto the image sensor 103 when the subject described above is far away. A spherical wave from a point 1701 constituting a distant object becomes a plane wave while propagating a sufficiently long distance and irradiates the imaging pattern 105, and when the projected image 1702 is projected onto the image sensor 103, the projected image is The shape is almost the same as that of the photographing pattern 105. As a result, it is possible to obtain a single bright spot by performing development processing on the projection image 1702 using the development pattern.

  On the other hand, imaging for a finite distance object will be described. FIG. 18 is a diagram illustrating that the shooting pattern 105 is enlarged when the object is at a finite distance. When the spherical wave from the point 1801 constituting the object irradiates the photographing pattern 105 and the projected image 1802 is projected onto the image sensor 103, the projected image is enlarged substantially uniformly. The enlargement ratio α is obtained by using the distance f from the shooting pattern 105 to the point 1801.

のように算出できる。そのため、平行光に対して設計された現像用パターンをそのまま用いて現像処理したのでは、単一の輝点を得ることができない。そこで、一様に拡大された撮影用パターン１０５の投影像に合わせて、現像用パターン８０１を拡大させたならば、拡大された投影像１８０２に対して再び、単一の輝点を得ることができる。このためには、現像用パターン８０１の係数βをβ／α^２とすることで補正が可能である。これにより、必ずしも無限遠でない距離の点１８０１からの光を選択的に再生することができる。これによって、任意の位置に焦点合わせて撮影を行うことができる。

It can be calculated as follows. Therefore, a single bright spot cannot be obtained if the development pattern designed for parallel light is used as it is. Therefore, if the development pattern 801 is enlarged in accordance with the projected image of the photographing pattern 105 that is uniformly enlarged, a single bright spot can be obtained again with respect to the enlarged projected image 1802. it can. For this purpose, correction can be performed by setting the coefficient β of the development pattern 801 to β / α ² . Thereby, light from the point 1801 at a distance that is not necessarily infinite can be selectively reproduced. As a result, it is possible to perform shooting while focusing on an arbitrary position.

  Furthermore, according to this configuration, it is possible to focus on an arbitrary distance after shooting. FIG. 19 is a diagram illustrating a configuration example of the imaging apparatus 101 having a focus function. Unlike FIG. 1, the imaging apparatus 101 includes a focus setting unit 1901. A focus setting unit 1901 can acquire a focus distance input by operating a hardware switch such as a knob provided in the imaging apparatus 101 or a GUI (Graphical User Interface) via the controller 108, and obtain focus distance information. The image is output to the image processing unit 107. The focus setting unit 1901 may be realized in the controller 108.

  Furthermore, the fact that focus adjustment after shooting is possible in this way means that depth information is included, and various functions such as autofocus and distance measurement can be realized in the image processing unit 107. . In order to realize such functions as focus adjustment, it is necessary to freely change the coefficient β of the development pattern 801. However, like the processing in the fringe scan processing unit 106 described in the present embodiment. In addition, by performing the fringe scan calculation using only the sensor image, it is possible to perform the image processing using the development pattern 801 independently, and the processing can be simplified. In other words, when focus adjustment or the like is performed after imaging, the amount of calculation can be greatly reduced because the development pattern is calculated for the complex sensor image without performing the fringe scan again in the image processing unit.

  According to the method and configuration according to the present embodiment, it is possible to realize an imaging apparatus capable of separately performing focus adjustment, autofocus, and ranging image processing after imaging and fringe scanning processing. . In other words, when focus adjustment or the like is performed after imaging, the amount of calculation can be greatly reduced because the development pattern is calculated for the complex sensor image without performing the fringe scan again in the image processing unit.

[Second Embodiment]
In the imaging apparatus 101 according to the first embodiment, processing by the image processing unit 107 becomes heavy due to enhancement of functions such as focus adjustment, autofocus, and distance measurement, and further, the size, cost, and power consumption of the imaging apparatus 101 increase. There is a possibility that. Therefore, in the second embodiment, a process division method for reducing at least one of processing, size, cost, power consumption, and the like of the imaging apparatus 101 will be described. Hereinafter, a description will be given focusing on differences from the first embodiment.

  FIG. 20 is a diagram illustrating a configuration example of an imaging system according to the second embodiment. The imaging system is obtained by dividing the imaging apparatus 101 of the first embodiment into an imaging apparatus 2001 that is a transmission side apparatus and an image processing apparatus 2002 that is a reception side apparatus. In this imaging system, the imaging device 2001 and the image processing device 2002 transmit and receive data via a wired or wireless communication method or a combination thereof, or a storage medium. The communication means can be composed of one or more combinations of various communication networks such as a LAN (Local Area Network) and the Internet. In this case, the image processing apparatus 2002 can be realized by a server computer, for example, and the server computer may be able to communicate with a plurality of imaging apparatuses 2001.

  The imaging apparatus 2001 includes a data transmission unit 2003 in addition to the imaging unit 102 and the fringe scan processing unit 106. The data transmission unit 2003 converts the complex sensor image output from the fringe scan processing unit 106 into a format to be transmitted to a predetermined communication network or storage medium and transmits the complex sensor image. The data transmission unit 2003 may be, for example, an interface that conforms to a wired or wireless LAN communication standard, an interface that conforms to a mobile communication network communication standard, or a USB (Universal Serial Bus). An interface conforming to a communication standard such as The imaging device 2001 may include a plurality of data transmission units 2003 with different communication standards, and may be used properly according to the communication environment.

  The image processing apparatus 2002 includes a data receiving unit 2004 in addition to the image processing unit 107 and the controller 108. The data receiving unit 2004 receives data (complex sensor image) transmitted from the imaging device 2001, converts the data into a predetermined format handled by the image processing unit 107, and outputs the data. The data reception unit 2004 is an interface similar to the data transmission unit 2003 described above. The image processing unit 107 uses the complex sensor image output from the data receiving unit 2004 to realize functions such as focus adjustment, autofocus, and distance measurement.

  According to the method and configuration according to the present embodiment, by transmitting a sensor image to an external device, the configuration of the imaging device 2001 can be simplified, and a small size, light weight, and low cost can be realized. Further, by transmitting the complex sensor image after the fringe scan calculation to an external device, it is possible to realize high functions such as focus adjustment, autofocus, and distance measurement with an image processing device 2002 that is faster than the imaging device 2001, for example.

[Third Embodiment]
In the imaging system of the second embodiment, the amount of communication data between the imaging device 2001 and the image processing device 2002 is large, and there is a possibility that the transmission band and power consumption necessary for communication will increase. Therefore, in the third embodiment, a data amount reduction method for reducing the communication data amount will be described. Hereinafter, a description will be given focusing on differences from the second embodiment.

  First, a method for reducing the data amount of the sensor image will be described. FIG. 21 shows an example of a sensor image that is an output of the image sensor 103 when a certain subject is photographed. FIG. 22 is a diagram showing an example of the luminance distribution of the sensor image, and shows the result of mapping the luminance of each pixel position on the straight line connecting AB in FIG. As described above, the sensor image has a signal component concentrated near the center of the range with respect to the sensor dynamic range. Specifically, only the data within the range of the maximum and minimum values of the signal component is extracted, and other data is reduced, thereby reducing the number of necessary bits (for example, reducing the upper bits). Thus, the data amount can be reduced.

  However, depending on the shooting conditions, as shown in FIG. 23 (a diagram showing another example of the luminance distribution of the sensor image), luminance unevenness may occur on the sensor, and the signal component may be distributed over the entire sensor dynamic range. In this case, the data reduction efficiency is deteriorated.

  Therefore, the fringe scan calculation in the fringe scan processing unit 106 is effective. For example, in the fringe scan using four phases as shown in FIG. 12, phases of Φ = 0, π / 2, π, 3π / 2 are used, and Φ = 0 and π, and Φ = π / A phase having a phase difference π, such as 2 and 3π / 2, is a pattern in which transmission and non-transmission are reversed. That is, since the luminance unevenness is the same and the pattern is inverted, if subtraction is performed between sensor images having this relationship, as shown in FIG. 24 (a diagram showing an example of the luminance distribution of a complex sensor image), the luminance unevenness It is possible to obtain a signal component in which the average value is substantially zero by removing. Thereby, it is possible to always limit the signal component to data within a predetermined range, and the data amount reduction processing can be efficiently performed. Note that the subtraction results of Φ = 0 and π and Φ = π / 2 and 3π / 2 are the real part of the complex sensor image that is the output of the fringe scan processing unit 106 and the subtraction result of Φ = π / 2 and 3π / 2. It is an imaginary part.

Next, a method for reducing the data amount of the complex sensor image will be described in detail. FIG. 25 shows an example in which complex sensor image data is mapped onto a complex space. As described above, the complex sensor image after the fringe scan calculation is distributed in the complex space. FIG. 26 shows an example of a histogram of real part data of the complex sensor image. It is only necessary to calculate the maximum value (I _max ) and the minimum value (I _min ) of this distribution 2601 and output only real part data existing in this range. The same applies to the imaginary part data of the complex sensor image.

  In view of the fact that the sensor image includes pulse-like noise, there is a possibility that the data amount cannot be sufficiently reduced only by outputting the data in the range of the maximum value and the minimum value. Further, in this embodiment, since the data after development is considered to be scattered over the entire surface of the sensor, even if a part of the data of the sensor image is insufficient, the quality after development is not greatly affected. Therefore, in the distribution 2601 of FIG. 26, a method of not outputting all the distributed data, such as outputting data in a range within the standard deviation σ of the distribution centering on 0, or outputting data in the range of 3σ. Also good. Thereby, the amount of data can be further reduced.

  Further, the output data may be limited to a preset range / accuracy instead of the above-described maximum and minimum values or standard deviation. For example, when limiting to a preset range, when the number of bits of the complex sensor image is 13 bits, the upper 5 bits are always reduced and output as 8 bits. For example, when limiting to a preset accuracy, when the number of bits of the complex sensor image is 13 bits, the lower 5 bits are always reduced or rounded and output as 8 bits. Thereby, there is no need to calculate the maximum value, the minimum value, the standard deviation, etc., and the amount of calculation can be reduced.

  In addition, when processing moving images, the maximum value, minimum value, standard deviation, etc. described above are determined by using the results of frames processed before the frame being processed, and the output data is limited. Also good. Thereby, it is possible to suppress the influence due to the delay of the arithmetic processing and efficiently reduce the data amount.

  FIG. 27 shows another example in which complex sensor image data is mapped onto a complex space. FIG. 28 shows an example of a histogram of real part data of the complex sensor image. Thus, there is a possibility that the average value μ0 of the distribution 2801 is not zero. In this case, μ0 may be subtracted from the data before outputting. In the present embodiment, since the image processing unit 107 performs development processing mainly including Fourier transform, the luminance offset does not directly affect the developed image. If there is an influence, there is a possibility that the DC component of the frequency spectrum in the moire development method has a peak, but this can be solved by masking the corresponding pixel.

25 and 27 describe an example having an isotropic distribution in a complex space, but depending on the shooting situation, the distribution may be biased as shown in FIGS. 29 and 30. 29 and 30 show still another example in which the data of the complex sensor image is mapped on the complex space. In the anisotropic distribution as shown in FIG. 29, when the average value θ ₀ of the declination is 0, the output data may be limited in different ranges between the real part and the imaginary part. If θ ₀ is not 0, it is effective to convert the complex sensor image into polar coordinates. After the polar coordinate conversion, the output data may be limited within a range different between the real part and the imaginary part by multiplying by the declination angle (−θ ₀ ). Alternatively, the range / accuracy of the phase data of the complex sensor image may be limited, and polar coordinate amplitude / phase data may be output. Similarly, in the annular distribution as shown in FIG. 30, it is effective to convert the complex sensor image into polar coordinates. Limit the scope and accuracy of the amplitude data around the average value r ₀ of the amplitude of the complex sensor image may output the data of polar coordinates of amplitude and phase.

  Furthermore, when the distribution bias as shown in FIG. 29 is extreme, a method of outputting only one of the real part and the imaginary part, or only one of the amplitude and the phase may be employed. As a result, the data amount can be significantly reduced.

  A configuration for performing the above data reduction processing will be described. FIG. 31 is a diagram illustrating a configuration example of an imaging system according to the third embodiment. A difference from the imaging system (FIG. 20) of the second embodiment is an imaging apparatus 3101 that replaces the imaging apparatus 2001. Unlike the imaging apparatus 2001, the imaging apparatus 3101 includes a data amount reduction unit 3102. In the imaging apparatus 3101, the fringe scan processing unit 106 outputs a complex sensor image, the data amount reduction unit 3102 reduces the data amount by reducing the number of bits of the complex sensor image, and the data transmission unit 2003 reduces the data amount. The obtained complex sensor image is transmitted to an external device.

  FIG. 32 is a flowchart illustrating a processing example of the data amount reduction unit 3102. First, the data amount reduction unit 3102 acquires the complex sensor image output from the fringe scan processing unit 106, and subtracts the average value μ0 if the average value μ0 is not 0 (3201). Next, the data amount reduction unit 3102 separates the complex sensor image into an imaginary part and a real part (3202). Subsequently, the data amount reduction unit 3102 determines the output data range by obtaining the maximum value and the minimum value for each of the imaginary part and the real part (3203), and after determining the bit precision necessary for the range ( 3204), the number of bits of the output data is limited within the determined range / accuracy and output (3205).

  FIG. 33 is a flowchart illustrating another example of processing of the data amount reduction unit 3102. FIG. 33 differs from the processing of FIG. 32 in that polar coordinates are converted and output. First, the data amount reduction unit 3102 acquires the complex sensor image output from the fringe scan processing unit 106, and subtracts the average value μ0 if the average value μ0 is not 0 (3201). Next, the data amount reduction unit 3102 converts the complex sensor image into polar coordinates, and then separates the complex sensor image into amplitude and phase (3301). Subsequently, the data amount reduction unit 3102 determines the output data range by obtaining the maximum value and the minimum value for each of the amplitude and phase (3302), and after determining the bit precision necessary for the range (3303). The number of bits of the output data is limited within the determined range / accuracy and output (3304). Thereby, it is possible to increase the data reduction efficiency when the distribution is anisotropic in the complex space.

FIG. 34 is a flowchart illustrating still another example of the processing of the data amount reduction unit 3102. The flowchart in FIG. 34 differs from FIG. 33 in that a process 3401 is added between the processes 3301 and 3302. The data amount reduction unit 3102 subtracts the phase average value θ ₀ and the amplitude average value r ₀ corresponding to the distribution shape from the separated amplitude / phase data (3401). Thereby, data reduction can be performed more efficiently.

  According to the method / configuration according to the present embodiment, the amount of data communication can be reduced by reducing the number of bits of sensor image data, and the transmission band and power consumption for network transmission can be reduced.

  The present invention has been described using a plurality of embodiments. Of course, the present invention is not limited to the above-described embodiment, and includes various modifications. For example, the above-described embodiment has been described in detail for easy understanding of the present invention, and is not necessarily limited to the one having all the configurations described.

  Further, a part of the configuration of an embodiment can be replaced with the configuration of another embodiment, and the configuration of another embodiment can be added to the configuration of an embodiment. In addition, it is possible to add, delete, and replace other configurations for a part of the configuration of each embodiment.

  Each of the above-described configurations, functions, processing units, processing means, and the like may be realized by hardware by designing a part or all of them with, for example, an integrated circuit. Each of the above-described configurations, functions, and the like may be realized by software by interpreting and executing a program that realizes each function by the processor. Information such as programs, tables, and files for realizing each function is stored in a memory, a hard disk, a recording device such as an SSD (Solid State Drive), or a recording medium such as an IC card, an SD card, or a DVD (Digital Versatile Disc). Can be put.

  Further, the control lines and information lines indicate what is considered necessary for the explanation, and not all the control lines and information lines on the product are necessarily shown. Actually, it may be considered that almost all the components are connected to each other.

  The present invention is not limited to an imaging device, an imaging method, and an image processing device, and can be provided in various modes such as an imaging system, an image processing method, and a computer-readable program.

DESCRIPTION OF SYMBOLS 101 ... Imaging device, 102 ... Imaging part, 103 ... Image sensor, 103a ... Pixel, 104 ... Pattern substrate, 105 ... Imaging pattern, 106 ... Fringe scan processing part, 107 ... Image processing part, 108 ... Controller, 301 ... Support 801 ... Development pattern, 1701 ... Point, 1702 ... Projected image, 1801 ... Point, 1802 ... Projected image, 1901 ... Focus setting unit, 2001 ... Imaging device, 2002 ... Image processing device, 2003 ... Data transmission unit, 2004 ... Data receiver, 2601 ... distribution, 2801 ... distribution, 3101 ... imaging device, 3102 ... data amount reduction part

Claims (23)

A modulator that modulates the intensity of light based on a grating pattern;
An image sensor that converts the light transmitted through the modulator into an electrical signal to generate a sensor image;
A complex sensor image processing unit for generating a complex sensor image having a complex number from the sensor image;
An image pickup apparatus comprising: a data transmission unit that transmits the complex sensor image. The imaging apparatus according to claim 1,
The imaging apparatus, wherein the complex sensor image processing unit generates the complex sensor image from at least two sensor images having different phase patterns. The imaging apparatus according to claim 1,
The complex sensor image processing unit generates the complex sensor image based on the following expression.

The imaging apparatus according to claim 1,
An image pickup apparatus comprising: a data amount reducing unit that reduces the data amount of the complex sensor image. The imaging apparatus according to claim 4,
The data amount reducing unit reduces the data amount by separating the complex sensor image into a real part and an imaginary part. The imaging apparatus according to claim 4,
The data amount reduction unit reduces the data amount by separating the complex sensor image into an amplitude and a phase. The imaging apparatus according to claim 4,
The data amount reducing unit determines a data range and accuracy based on a maximum value and a minimum value of the complex sensor image, and reduces the number of output bits. The imaging apparatus according to claim 4,
The data amount reduction unit determines a data range and accuracy based on a standard deviation of the distribution of the complex sensor image, and reduces the number of output bits. An image processing device that communicates with an imaging device,
The imaging apparatus includes: a modulator that modulates light intensity based on a first grating pattern; an image sensor that converts light transmitted through the modulator into an electrical signal to generate a sensor image; and A complex sensor image processing unit that generates a complex sensor image having a complex number; and a data transmission unit that transmits the complex sensor image;
The image processing apparatus includes:
A data receiver for receiving the complex sensor image;
An image processing apparatus comprising: an image processing unit that restores an image based on a calculation of the complex sensor image and data of a second lattice pattern. The image processing apparatus according to claim 9,
The complex sensor image processing unit generates the complex sensor image from at least two sensor images having different phase patterns. The image processing apparatus according to claim 9,
The complex sensor image processing unit generates the complex sensor image based on the following expression.

The image processing apparatus according to claim 9,
The image processing apparatus includes a data amount reducing unit that reduces a data amount of the complex sensor image. The image processing apparatus according to claim 12,
The data amount reducing unit reduces the data amount by separating the complex sensor image into a real part and an imaginary part. The image processing apparatus according to claim 12,
The data amount reduction unit reduces the data amount by separating the complex sensor image into an amplitude and a phase. The image processing apparatus according to claim 12,
The data amount reducing unit determines a data range and accuracy based on a maximum value and a minimum value of the complex sensor image, and reduces the number of output bits. The image processing apparatus according to claim 12,
The data amount reduction unit determines a data range and accuracy based on a standard deviation of the distribution of the complex sensor image, and reduces the number of output bits. A modulator that modulates the intensity of the light based on the first grating pattern;
An image sensor that converts the light transmitted through the modulator into an electrical signal to generate a sensor image;
A complex sensor image processing unit for generating a complex sensor image having a complex number from the sensor image;
An imaging apparatus comprising: an image processing unit that restores an image based on a calculation of the complex sensor image and data of a second lattice pattern. The imaging device according to claim 17,
The complex sensor image processing unit generates the complex sensor image from at least two sensor images having different phase patterns. The imaging device according to claim 17,
The complex sensor image processing unit generates the complex sensor image based on the following expression.

A modulation step for modulating the intensity of the light based on the first grating pattern;
An image generation step of converting the modulated light into an electrical signal to generate a sensor image;
A complex sensor image processing step of generating a complex sensor image having a complex number from the sensor image;
And a data transmission step of transmitting the complex sensor image. The imaging method according to claim 20, wherein
The complex sensor image processing step generates the complex sensor image from at least two sensor images having different phase patterns. The imaging method according to claim 20, wherein
An imaging method, wherein the complex sensor image is generated in the complex sensor image processing step based on the following equation.

The imaging method according to claim 20, wherein
An image processing method comprising: an image processing step of restoring an image based on a calculation of the complex sensor image and data of a second lattice pattern.

Images